Oct 31 12:56:54.099545 ip-10-0-137-206 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 12:56:54.099559 ip-10-0-137-206 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 12:56:54.099569 ip-10-0-137-206 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 12:56:54.099914 ip-10-0-137-206 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 12:57:04.222582 ip-10-0-137-206 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 12:57:04.222601 ip-10-0-137-206 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d8f72d01f47d457b98c00265a10b5dde -- Oct 31 12:59:03.164735 ip-10-0-137-206 systemd[1]: Starting Kubernetes Kubelet... Oct 31 12:59:03.587650 ip-10-0-137-206 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:59:03.587650 ip-10-0-137-206 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 12:59:03.587650 ip-10-0-137-206 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:59:03.587650 ip-10-0-137-206 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:59:03.593202 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593126 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:03.593202 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593202 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593207 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593210 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593213 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593216 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593219 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593222 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593224 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593227 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593230 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593232 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593235 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593238 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593241 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593243 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593246 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593249 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593252 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593255 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593257 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:03.593279 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593260 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593263 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593266 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593268 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593271 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593273 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593276 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593279 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593282 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593286 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593288 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593291 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593294 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593297 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593300 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593302 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593305 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593308 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593312 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593314 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:03.593757 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593317 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593319 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593322 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593325 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593328 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593330 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593333 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593335 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593338 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593341 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593343 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593346 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593349 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593351 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593354 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593357 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593359 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593362 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593365 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:03.594268 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593367 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593370 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593374 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593378 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593382 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593385 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593387 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593390 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593394 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593398 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593401 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593403 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593406 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593408 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593411 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593413 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593416 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593419 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593422 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:03.594734 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593424 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593426 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593429 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593432 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593434 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593437 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593439 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593442 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593445 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593447 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593450 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593452 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593455 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593458 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593461 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593463 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593466 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593469 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593471 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593490 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:03.595188 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593493 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593495 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593499 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593501 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593504 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593507 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.593509 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594261 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594268 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594271 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594275 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594278 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594282 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594284 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594287 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594290 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594292 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594295 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594298 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594300 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:03.595743 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594304 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594307 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594310 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594312 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594315 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594318 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594321 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594323 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594326 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594329 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594331 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594334 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594337 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594339 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594342 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594344 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594347 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594349 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594352 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594354 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:03.596221 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594357 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594360 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594362 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594366 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594368 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594371 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594373 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594376 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594378 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594381 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594384 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594386 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594389 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594391 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594394 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594397 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594400 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594403 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594405 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:03.596729 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594408 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594411 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594413 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594416 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594418 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594421 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594425 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594430 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594433 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594436 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594439 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594442 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594445 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594447 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594449 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594452 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594455 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594458 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594461 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:03.597176 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594463 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594466 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594468 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594471 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594489 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594492 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594495 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594497 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594500 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594503 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594506 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594508 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594511 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594514 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594516 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594519 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594521 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594524 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594526 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594529 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:03.597664 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594531 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594534 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594536 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594540 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594543 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594546 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594550 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594552 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594555 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594557 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594560 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594563 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594566 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594568 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.594571 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595024 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595029 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595032 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595036 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:03.598168 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595039 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595041 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595044 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595047 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595050 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595052 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595055 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595057 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595060 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595063 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595065 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595068 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595070 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595073 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595075 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595078 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595081 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595083 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595086 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:03.598627 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595088 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595091 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595094 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595096 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595099 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595102 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595104 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595107 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595109 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595112 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595115 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595117 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595120 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595123 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595125 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595128 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595130 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595133 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595136 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:03.599070 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595138 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595141 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595143 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595146 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595148 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595151 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595153 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595155 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595158 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595161 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595163 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595166 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595168 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595171 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595173 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595176 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595178 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595183 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595186 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595188 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:03.599566 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595192 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595194 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595199 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595202 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595205 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595208 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595211 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595214 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595216 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595219 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595221 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595224 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595226 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595229 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595231 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595233 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595238 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595241 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595245 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:03.600047 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595248 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595251 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595254 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595256 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595259 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595261 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595264 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595267 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595269 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595272 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595275 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595278 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595281 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595284 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595286 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595290 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595293 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595296 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595298 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595301 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:03.600515 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595304 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595307 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595310 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595312 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.595315 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595402 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595408 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595414 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595421 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595427 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595432 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595438 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595445 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595448 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595451 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595455 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595458 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595461 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595465 2583 flags.go:64] FLAG: --cgroup-root="" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595468 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595471 2583 flags.go:64] FLAG: --client-ca-file="" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595485 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595490 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595494 2583 flags.go:64] FLAG: --cluster-domain="" Oct 31 12:59:03.601030 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595497 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595500 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595504 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595509 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595513 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595516 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595519 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595522 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595525 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595528 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595532 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595535 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595538 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595542 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595545 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595549 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595552 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595555 2583 flags.go:64] FLAG: --enable-server="true" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595558 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595562 2583 flags.go:64] FLAG: --event-burst="100" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595565 2583 flags.go:64] FLAG: --event-qps="50" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595568 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595572 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595574 2583 flags.go:64] FLAG: --eviction-hard="" Oct 31 12:59:03.601629 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595579 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595582 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595585 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595589 2583 flags.go:64] FLAG: --eviction-soft="" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595592 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595595 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595599 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595602 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595605 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595608 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595610 2583 flags.go:64] FLAG: --feature-gates="" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595616 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595620 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595624 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595627 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595630 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595633 2583 flags.go:64] FLAG: --help="false" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595636 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595639 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595643 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595645 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595649 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595652 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 12:59:03.602221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595655 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595658 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595661 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595664 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595667 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595670 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595674 2583 flags.go:64] FLAG: --kube-reserved="" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595676 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595679 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595682 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595685 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595688 2583 flags.go:64] FLAG: --lock-file="" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595691 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595694 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595697 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595704 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595707 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595710 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595713 2583 flags.go:64] FLAG: --logging-format="text" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595716 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595721 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595725 2583 flags.go:64] FLAG: --manifest-url="" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595728 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595732 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595735 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 12:59:03.602784 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595740 2583 flags.go:64] FLAG: --max-pods="110" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595743 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595746 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595749 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595752 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595755 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595758 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595761 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595769 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595772 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595775 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595778 2583 flags.go:64] FLAG: --pod-cidr="" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595781 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595784 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595787 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595790 2583 flags.go:64] FLAG: --port="10250" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595794 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595796 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08340c8f5e8ff6077" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595800 2583 flags.go:64] FLAG: --qos-reserved="" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595803 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595806 2583 flags.go:64] FLAG: --register-node="true" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595809 2583 flags.go:64] FLAG: --register-with-taints="" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595813 2583 flags.go:64] FLAG: --registry-burst="10" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595816 2583 flags.go:64] FLAG: --registry-qps="5" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595819 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595822 2583 flags.go:64] FLAG: --reserved-memory="" Oct 31 12:59:03.603366 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595826 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595830 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595833 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595836 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595840 2583 flags.go:64] FLAG: --runonce="false" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595843 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595846 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595849 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595852 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595855 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595858 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595861 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595864 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595867 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595869 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595872 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595875 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595878 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595881 2583 flags.go:64] FLAG: --system-cgroups="" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595884 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595889 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595892 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595895 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595898 2583 flags.go:64] FLAG: --tls-min-version="" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595901 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 12:59:03.603982 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595904 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595907 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595910 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595914 2583 flags.go:64] FLAG: --v="2" Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595918 2583 flags.go:64] FLAG: --version="false" Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595922 2583 flags.go:64] FLAG: --vmodule="" Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595926 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.595930 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596054 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596058 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596060 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596064 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596066 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596069 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596072 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596074 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596077 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596080 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596082 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596085 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596088 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596090 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:03.604622 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596093 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596096 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596098 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596101 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596104 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596107 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596109 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596112 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596114 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596117 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596120 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596122 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596125 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596127 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596130 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596133 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596135 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596138 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596141 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:03.605148 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596144 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596147 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596149 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596153 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596155 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596158 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596160 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596163 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596166 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596168 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596171 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596174 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596176 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596179 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596182 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596184 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596187 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596190 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596192 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:03.605615 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596195 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596198 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596200 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596203 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596205 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596208 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596210 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596213 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596216 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596219 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596222 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596225 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596229 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596232 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596235 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596238 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596240 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596244 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596247 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596249 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:03.606078 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596252 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596255 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596259 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596263 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596267 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596269 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596272 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596275 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596277 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596280 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596284 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596286 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596289 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596291 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596294 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596296 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596299 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596302 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596304 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:03.606629 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596308 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596311 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596314 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596316 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596319 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596322 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596325 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596328 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596330 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596333 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596336 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596338 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596341 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596344 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596347 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.596372 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:59:03.607075 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596409 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596414 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596417 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596419 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596423 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596426 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596428 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596434 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596437 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596439 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596442 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596444 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596447 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596450 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596452 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596455 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596457 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596460 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596463 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:03.607471 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596465 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596468 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596470 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596487 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596490 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596492 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596495 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596498 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596501 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596504 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596507 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596510 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596512 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596515 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596518 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596521 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596523 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596526 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596528 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:03.607923 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596531 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596535 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596538 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596541 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596543 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596546 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596549 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596551 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596554 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596557 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596560 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596564 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596566 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596569 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596572 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596574 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596578 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596581 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596583 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596586 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:03.608380 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596589 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596592 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596596 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596600 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596603 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596606 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596609 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596612 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596614 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596617 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596620 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596623 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596625 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596629 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596632 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596634 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596637 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596640 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596643 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:03.608866 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596647 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596650 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596653 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596655 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596658 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596661 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596663 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596666 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596668 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596673 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596675 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596678 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596681 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596684 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596686 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596689 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596691 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596695 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596697 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596700 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:03.609313 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596702 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596705 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596708 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596710 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596713 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596716 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596719 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596722 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.596724 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.604515 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.604530 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.605948 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.605954 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606030 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606035 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606039 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606042 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606044 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606047 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606050 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:03.609890 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606053 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606055 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606059 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606062 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606065 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606067 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606070 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606073 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606076 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606080 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606084 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606087 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606090 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606093 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606096 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606098 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606101 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606104 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606106 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:03.610355 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606109 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606112 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606114 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606124 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606127 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606129 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606132 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606135 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606138 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606140 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606143 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606147 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606151 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606155 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606158 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606161 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606164 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606167 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606170 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:03.610814 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606173 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606175 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606178 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606180 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606183 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606186 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606189 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606191 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606194 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606197 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606199 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606202 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606205 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606207 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606211 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606213 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606216 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606225 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606229 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606231 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:03.611264 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606234 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606237 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606239 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606242 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606244 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606247 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606249 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606252 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606254 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606257 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606259 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606262 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606265 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606267 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606270 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606273 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606279 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606281 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606284 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606287 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:03.611741 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606289 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606293 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606295 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606298 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606300 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606303 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606306 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606309 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606311 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606314 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606322 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606325 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606327 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606330 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606333 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606335 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606338 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606341 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606343 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:03.612203 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606346 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606348 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.606382 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606587 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606592 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606595 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606597 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606601 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606603 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606606 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606608 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606611 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606614 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606616 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606619 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606622 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606624 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:03.612681 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606627 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606630 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606633 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606635 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606638 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606641 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606644 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606653 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606656 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606658 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606661 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606664 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606666 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606669 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606671 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606674 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606676 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606679 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606681 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:03.613147 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606684 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606686 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606689 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606692 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606694 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606696 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606699 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606702 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606704 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606707 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606710 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606713 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606717 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606720 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606723 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606725 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606728 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606730 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606733 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:03.613667 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606735 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606738 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606746 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606749 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606751 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606754 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606756 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606759 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606761 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606764 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606766 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606769 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606771 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606774 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606776 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606779 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606781 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606784 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606786 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606789 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:03.614108 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606791 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606794 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606796 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606799 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606802 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606806 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606809 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606813 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606815 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606818 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606821 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606824 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606826 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606829 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606831 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606839 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606842 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606845 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606847 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:03.614609 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606851 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606853 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606856 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606858 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606861 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606864 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606866 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606869 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606871 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606874 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606876 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606879 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606881 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606884 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606886 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.606910 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:59:03.615044 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606947 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606951 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606954 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606957 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606960 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606963 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606965 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606968 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606971 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606974 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606976 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606979 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606982 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606991 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606994 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606997 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.606999 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607002 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607004 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607007 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:03.615418 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607009 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607012 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607014 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607017 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607019 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607022 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607024 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607027 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607029 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607032 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607034 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607037 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607040 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607042 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607045 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607047 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607050 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607052 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607055 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607058 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:03.615921 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607060 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607063 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607065 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607068 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607071 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607073 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607082 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607085 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607087 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607090 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607093 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607095 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607098 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607100 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607103 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607106 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607108 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607111 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607113 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:03.616441 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607116 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607118 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607121 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607123 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607126 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607129 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607131 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607135 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607139 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607142 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607145 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607148 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607150 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607153 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607155 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607158 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607160 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607163 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607165 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:03.617008 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607168 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607178 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607180 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607183 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607185 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607188 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607190 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607193 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607195 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607198 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607200 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607203 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607205 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607208 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607210 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607213 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607217 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607220 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607222 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607225 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:03.617529 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607228 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607232 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607236 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607239 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607241 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607244 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607246 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:03.607250 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.607310 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.611158 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.612122 2583 server.go:1014] "Starting client certificate rotation" Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.612225 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.612264 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:59:03.618000 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.617032 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:59:03.619310 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.619290 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:59:03.630130 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.630112 2583 log.go:25] "Validated CRI v1 runtime API" Oct 31 12:59:03.633691 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.633680 2583 log.go:25] "Validated CRI v1 image API" Oct 31 12:59:03.634570 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.634543 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 12:59:03.640451 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.640431 2583 fs.go:135] Filesystem UUIDs: map[300f6a1f-be2a-4cdb-9abf-9a5901ab94c2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 aabf960d-ae26-4290-aea0-4b954e44ae1a:/dev/nvme0n1p4] Oct 31 12:59:03.640520 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.640449 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 12:59:03.645714 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.645569 2583 manager.go:217] Machine: {Timestamp:2026-10-31 12:59:03.644138068 +0000 UTC m=+0.374751921 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089803 MemoryCapacity:33164320768 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25468284c75cf359c37b4b40288bf2 SystemUUID:ec254682-84c7-5cf3-59c3-7b4b40288bf2 BootID:d8f72d01-f47d-457b-98c0-0265a10b5dde Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582160384 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:58:22:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:58:22:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:f5:ec:3a:1d:c7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164320768 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 12:59:03.645714 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.645713 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 12:59:03.645829 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.645801 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 12:59:03.648485 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.648452 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 12:59:03.648617 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.648488 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-206.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 12:59:03.648667 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.648618 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 12:59:03.648667 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.648627 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 12:59:03.648667 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.648658 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:59:03.648742 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.648676 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:59:03.649870 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.649856 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 12:59:03.653536 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.653520 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 12:59:03.654191 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.654179 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 12:59:03.654312 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.654302 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 12:59:03.654342 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.654336 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 31 12:59:03.654376 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.654347 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 12:59:03.654376 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.654357 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 12:59:03.654376 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.654365 2583 kubelet.go:405] "Adding apiserver pod source" Oct 31 12:59:03.654376 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.654375 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 12:59:03.654719 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.654703 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:59:03.656966 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.656947 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 12:59:03.666906 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.666621 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:59:03.668181 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.668167 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 12:59:03.668230 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.668199 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 12:59:03.668386 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.668375 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 12:59:03.668438 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.668391 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 12:59:03.668438 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.668398 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 12:59:03.668438 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.668404 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 12:59:03.668438 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.668410 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 12:59:03.668438 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.668415 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 12:59:03.669419 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.669409 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 12:59:03.669449 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.669419 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 12:59:03.669449 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.669427 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 12:59:03.669449 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.669433 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 12:59:03.669564 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.669450 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 12:59:03.669564 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.669459 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 12:59:03.670087 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.670078 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 12:59:03.670127 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.670091 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 12:59:03.672922 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.672903 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-206.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 12:59:03.673929 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.673916 2583 server.go:1263] "Started kubelet" Oct 31 12:59:03.674016 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.673993 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 12:59:03.674086 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.673988 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 12:59:03.674134 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.674090 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 12:59:03.674656 ip-10-0-137-206 systemd[1]: Started Kubernetes Kubelet. Oct 31 12:59:03.676911 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.676728 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 31 12:59:03.677209 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.677163 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 12:59:03.677329 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.677314 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wzvtw" Oct 31 12:59:03.680508 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.680469 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 12:59:03.681881 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.681856 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 12:59:03.682378 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.682365 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 12:59:03.682640 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.682513 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:03.683038 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683021 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 12:59:03.683118 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683028 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 12:59:03.683118 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683041 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 12:59:03.683222 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683122 2583 factory.go:55] Registering systemd factory Oct 31 12:59:03.683222 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683153 2583 factory.go:223] Registration of the systemd container factory successfully Oct 31 12:59:03.683222 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683124 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 31 12:59:03.683222 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683191 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 31 12:59:03.683388 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683336 2583 factory.go:153] Registering CRI-O factory Oct 31 12:59:03.683388 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683346 2583 factory.go:223] Registration of the crio container factory successfully Oct 31 12:59:03.683468 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683396 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 12:59:03.683468 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683414 2583 factory.go:103] Registering Raw factory Oct 31 12:59:03.683468 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.683423 2583 manager.go:1199] Started watching for new ooms in manager Oct 31 12:59:03.684355 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.684339 2583 manager.go:319] Starting recovery of all containers Oct 31 12:59:03.689369 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.689340 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 12:59:03.689469 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.689392 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-206.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 12:59:03.689675 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.689613 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-206.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 12:59:03.690604 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.689400 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-206.us-east-2.compute.internal.18e39ead468ee4ca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-206.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-206.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:59:03.673889994 +0000 UTC m=+0.404503847,LastTimestamp:2026-10-31 12:59:03.673889994 +0000 UTC m=+0.404503847,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-206.us-east-2.compute.internal,}" Oct 31 12:59:03.691134 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.691113 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wzvtw" Oct 31 12:59:03.694931 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.694902 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 12:59:03.697139 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.697102 2583 manager.go:324] Recovery completed Oct 31 12:59:03.703267 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.703254 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:59:03.705986 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.705969 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:59:03.706061 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.706002 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:59:03.706061 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.706018 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:59:03.706589 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.706573 2583 cpu_manager.go:226] "Starting" policy="none" Oct 31 12:59:03.706589 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.706588 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 12:59:03.706725 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.706604 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:59:03.706725 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.706607 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 12:59:03.707316 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.707302 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:59:03.707369 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.707332 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:59:03.707369 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.707357 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:59:03.708626 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.708556 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-206.us-east-2.compute.internal.18e39ead48789e64 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-206.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-206.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-206.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:59:03.705984612 +0000 UTC m=+0.436598470,LastTimestamp:2026-10-31 12:59:03.705984612 +0000 UTC m=+0.436598470,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-206.us-east-2.compute.internal,}" Oct 31 12:59:03.708693 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.708675 2583 policy_none.go:50] "Start" Oct 31 12:59:03.708693 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.708687 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 12:59:03.708753 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.708696 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 12:59:03.710612 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.710600 2583 policy_none.go:44] "Start" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.746801 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.754081 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.755301 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.755321 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.755353 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.755387 2583 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.759382 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.760329 2583 manager.go:341] "Starting Device Plugin manager" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.760357 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.760368 2583 server.go:87] "Starting device plugin registration server" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.760610 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.760618 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.760703 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.760772 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.760783 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.761339 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 12:59:03.764374 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.761385 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:03.856114 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.856062 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal"] Oct 31 12:59:03.856201 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.856118 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:59:03.856950 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.856933 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:59:03.857035 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.856961 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:59:03.857035 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.856971 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:59:03.859064 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.859051 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:59:03.859225 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.859209 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.859287 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.859247 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:59:03.859804 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.859779 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:59:03.859883 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.859813 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:59:03.859883 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.859826 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:59:03.859883 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.859786 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:59:03.859976 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.859884 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:59:03.859976 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.859895 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:59:03.860684 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.860672 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:59:03.861271 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.861256 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:59:03.861337 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.861285 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:59:03.861337 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.861296 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:59:03.861337 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.861315 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.861934 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.861921 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.861974 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.861946 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:59:03.862531 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.862517 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:59:03.862590 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.862542 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:59:03.862590 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.862551 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:59:03.870080 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.870068 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.870125 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.870088 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-206.us-east-2.compute.internal\": node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:03.884393 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.884372 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.884487 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.884396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.884487 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.884411 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df9f494cd77ee5cda911dbf4d5fd20aa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"df9f494cd77ee5cda911dbf4d5fd20aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.886562 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.886548 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:03.899741 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.899726 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" node="ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.902369 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.902354 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" node="ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.984804 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.984778 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.984935 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.984808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.984935 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.984830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df9f494cd77ee5cda911dbf4d5fd20aa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"df9f494cd77ee5cda911dbf4d5fd20aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.984935 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.984867 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.984935 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.984912 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df9f494cd77ee5cda911dbf4d5fd20aa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"df9f494cd77ee5cda911dbf4d5fd20aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.985075 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:03.984936 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:03.986895 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:03.986880 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:04.087800 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.087775 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:04.188514 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.188449 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:04.203618 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.203598 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:04.206752 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.206735 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:04.289332 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.289312 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:04.389800 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.389776 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:04.490358 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.490308 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:04.590784 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.590761 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:04.611928 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.611907 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 12:59:04.612032 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.612015 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 12:59:04.649281 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.649259 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:59:04.655278 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.655266 2583 apiserver.go:52] "Watching apiserver" Oct 31 12:59:04.665197 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.665179 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 12:59:04.665523 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.665508 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rwtn6","openshift-network-operator/iptables-alerter-l2brm","openshift-multus/network-metrics-daemon-sq2q6","openshift-network-diagnostics/network-check-target-sjsfh","openshift-ovn-kubernetes/ovnkube-node-pbk8s","kube-system/global-pull-secret-syncer-w58sk","openshift-dns/node-resolver-cs9pm","openshift-image-registry/node-ca-s7fjc","openshift-multus/multus-additional-cni-plugins-wprss","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw","openshift-cluster-node-tuning-operator/tuned-gvlv8","openshift-multus/multus-vgm78"] Oct 31 12:59:04.667792 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.667776 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:04.669831 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.669809 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:04.671887 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.671867 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:04.671968 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.671929 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:04.672221 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.672207 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 12:59:04.672276 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.672253 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 12:59:04.672276 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.672258 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5b9kz\"" Oct 31 12:59:04.673638 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.673576 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:59:04.673739 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.673658 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 12:59:04.673890 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.673865 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7lhn5\"" Oct 31 12:59:04.673978 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.673918 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 12:59:04.676465 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.676444 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:04.676584 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.676546 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:04.679305 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.679286 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.681413 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.681393 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:04.681517 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.681459 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:04.681991 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.681974 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 12:59:04.682934 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.682915 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:04.683578 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.683561 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:04.686073 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.686054 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:04.688868 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.688845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-hosts-file\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:04.688948 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.688893 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aaae2c3-0ea2-4061-a448-e50f82dcf823-agent-certs\") pod \"konnectivity-agent-rwtn6\" (UID: \"1aaae2c3-0ea2-4061-a448-e50f82dcf823\") " pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:04.688948 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.688919 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.689023 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.688922 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-netd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689076 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-config\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689132 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:04.689186 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689131 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-tmp-dir\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:04.689186 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-node-log\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689296 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689253 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-log-socket\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689296 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689280 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-host-slash\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:04.689397 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-env-overrides\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689397 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689377 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-iptables-alerter-script\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:04.689472 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689417 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-systemd-units\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689472 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689442 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-etc-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689544 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689472 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689544 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689506 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-script-lib\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689544 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689523 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgf48\" (UniqueName: \"kubernetes.io/projected/288087a8-c7ea-40a8-88bb-1b85be0aef55-kube-api-access-tgf48\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:04.689544 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689541 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmqn\" (UniqueName: \"kubernetes.io/projected/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-kube-api-access-ncmqn\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:04.689682 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-bin\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689682 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689604 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:04.689682 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689620 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689682 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689682 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689664 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-kubelet-config\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:04.689827 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689697 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-dbus\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:04.689827 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689713 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:04.689827 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689730 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9hhh\" (UniqueName: \"kubernetes.io/projected/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-kube-api-access-w9hhh\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:04.689827 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aaae2c3-0ea2-4061-a448-e50f82dcf823-konnectivity-ca\") pod \"konnectivity-agent-rwtn6\" (UID: \"1aaae2c3-0ea2-4061-a448-e50f82dcf823\") " pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:04.689827 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689767 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-netns\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689827 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689783 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-systemd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689827 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689797 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-var-lib-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689827 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689812 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-kubelet\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.689827 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689826 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-slash\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.690091 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-ovn\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.690091 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689859 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa9e011c-a51f-4738-b970-58ec77045b81-ovn-node-metrics-cert\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.690091 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.689874 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svj5d\" (UniqueName: \"kubernetes.io/projected/fa9e011c-a51f-4738-b970-58ec77045b81-kube-api-access-svj5d\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.691595 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.691546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.694875 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.694856 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.696068 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.696051 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:59:04.696144 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.696125 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:04.697248 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.697228 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal"] Oct 31 12:59:04.697336 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.697321 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.703535 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.703489 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:54:03 +0000 UTC" deadline="2028-04-22 22:58:47.875628895 +0000 UTC" Oct 31 12:59:04.703535 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.703532 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12945h59m43.1720989s" Oct 31 12:59:04.704248 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.704232 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:59:04.704313 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.704297 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal"] Oct 31 12:59:04.764857 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.764836 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 12:59:04.765013 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.764897 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fw57b" Oct 31 12:59:04.778670 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.778651 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fw57b" Oct 31 12:59:04.779517 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.779494 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 12:59:04.779517 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.779503 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 12:59:04.779841 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.779817 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 12:59:04.779922 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.779872 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:59:04.779985 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.779938 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 12:59:04.779985 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.779937 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 12:59:04.779985 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.779963 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 12:59:04.780139 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.780021 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 12:59:04.780139 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.780037 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 12:59:04.780220 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.780139 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 12:59:04.782225 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.780998 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 12:59:04.782225 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.781046 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drb8j\"" Oct 31 12:59:04.782225 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.781421 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 12:59:04.782225 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.781525 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2kvxk\"" Oct 31 12:59:04.782225 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.781705 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 12:59:04.782225 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.781839 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 12:59:04.782225 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.781953 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 12:59:04.782225 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.782084 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 12:59:04.782225 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.782108 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 12:59:04.782794 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.782310 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 12:59:04.783401 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.783383 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 12:59:04.783502 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.783485 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 12:59:04.784082 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.783658 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hwtqn\"" Oct 31 12:59:04.784082 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.783859 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2x4wq\"" Oct 31 12:59:04.784082 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.783866 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mw6wr\"" Oct 31 12:59:04.784082 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.783945 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 12:59:04.784082 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.784006 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2xlcn\"" Oct 31 12:59:04.784082 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.784081 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vtxsh\"" Oct 31 12:59:04.784376 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.784091 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 12:59:04.785185 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.785168 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 12:59:04.790078 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790046 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.790175 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790095 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-binary-copy\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.790175 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790127 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.790175 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790153 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d525238-6ced-4f9f-89a6-56fb50fba909-host\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:04.790324 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790172 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.790324 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-var-lib-kubelet\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.790324 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:04.790324 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790265 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aaae2c3-0ea2-4061-a448-e50f82dcf823-konnectivity-ca\") pod \"konnectivity-agent-rwtn6\" (UID: \"1aaae2c3-0ea2-4061-a448-e50f82dcf823\") " pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:04.790324 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790313 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-netns\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.790597 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790346 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-var-lib-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.790597 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790393 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.790597 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790420 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-netns\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.790597 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790447 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-kubelet\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.790597 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790530 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-kubelet\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.790597 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790587 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-slash\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.790914 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790619 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa9e011c-a51f-4738-b970-58ec77045b81-ovn-node-metrics-cert\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.790914 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790654 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svj5d\" (UniqueName: \"kubernetes.io/projected/fa9e011c-a51f-4738-b970-58ec77045b81-kube-api-access-svj5d\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.790914 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-host\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.790914 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790709 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-etc-tuned\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.790914 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790737 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-tmp\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.790914 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790825 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-system-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.790914 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790855 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-hosts-file\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:04.790914 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790892 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aaae2c3-0ea2-4061-a448-e50f82dcf823-konnectivity-ca\") pod \"konnectivity-agent-rwtn6\" (UID: \"1aaae2c3-0ea2-4061-a448-e50f82dcf823\") " pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:04.790914 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-config\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.791207 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790930 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-system-cni-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.791207 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790960 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wgc\" (UniqueName: \"kubernetes.io/projected/ef470cac-0cd7-4fae-87e2-f6d431227d00-kube-api-access-d7wgc\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.791207 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.790990 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mll9t\" (UniqueName: \"kubernetes.io/projected/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kube-api-access-mll9t\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.791207 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791012 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-lib-modules\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.791207 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-multus\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.791207 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-conf-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.791207 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791099 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-node-log\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.791207 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791124 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.791466 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791284 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 12:59:04.791466 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-run\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.791466 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-log-socket\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.791595 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791466 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-host-slash\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:04.791595 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791492 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwpx\" (UniqueName: \"kubernetes.io/projected/5d525238-6ced-4f9f-89a6-56fb50fba909-kube-api-access-jtwpx\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:04.791595 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791520 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-modprobe-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.791595 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791542 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d525238-6ced-4f9f-89a6-56fb50fba909-serviceca\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:04.791727 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791615 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cnibin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.791727 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791643 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-multus-certs\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.791727 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791670 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-device-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.791727 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791694 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-hosts-file\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:04.791727 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:04.791934 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791756 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.791934 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.791776 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:04.791934 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.791788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-kubelet-config\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:04.791934 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.791880 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs podName:288087a8-c7ea-40a8-88bb-1b85be0aef55 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:05.29186337 +0000 UTC m=+2.022477229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs") pod "network-metrics-daemon-sq2q6" (UID: "288087a8-c7ea-40a8-88bb-1b85be0aef55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:04.792064 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792017 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-node-log\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.792064 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792038 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.792130 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792061 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-log-socket\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.792130 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792100 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-host-slash\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:04.792210 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-dbus\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:04.792210 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792158 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.792210 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792183 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-etc-kubernetes\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.792352 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9hhh\" (UniqueName: \"kubernetes.io/projected/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-kube-api-access-w9hhh\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:04.792352 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792239 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-systemd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.792352 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792260 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-netns\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.792352 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792278 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-daemon-config\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.792352 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792338 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-kubelet-config\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:04.792506 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792449 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-dbus\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:04.792603 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792240 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-var-lib-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.792717 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792630 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-systemd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.792717 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792632 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-slash\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.792837 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792748 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-ovn\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.792837 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792788 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.792837 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792821 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-ovn\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.793005 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792833 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-config\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.793005 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-socket-dir-parent\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.793005 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792922 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw5pd\" (UniqueName: \"kubernetes.io/projected/7bf98082-9c6b-44e8-99e1-8ad8320b3483-kube-api-access-bw5pd\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.793005 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.792997 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aaae2c3-0ea2-4061-a448-e50f82dcf823-agent-certs\") pod \"konnectivity-agent-rwtn6\" (UID: \"1aaae2c3-0ea2-4061-a448-e50f82dcf823\") " pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:04.793204 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793033 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-netd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.793204 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793147 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:04.793204 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793158 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-netd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.793355 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.793231 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:04.793355 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.793277 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret podName:d83f70c7-ab6c-43b0-b86d-3d9d935973ec nodeName:}" failed. No retries permitted until 2026-10-31 12:59:05.293263814 +0000 UTC m=+2.023877656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret") pod "global-pull-secret-syncer-w58sk" (UID: "d83f70c7-ab6c-43b0-b86d-3d9d935973ec") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:04.793355 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793198 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-cnibin\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.793355 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793316 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-registration-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.793577 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-sys-fs\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.793577 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793374 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysconfig\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.793577 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793398 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-sys\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.793577 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793421 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-tmp-dir\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:04.793577 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793453 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-systemd\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.793577 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793469 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfpfl\" (UniqueName: \"kubernetes.io/projected/cabced46-3402-4172-aa1b-f056b9e01220-kube-api-access-xfpfl\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.793577 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793501 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-kubelet\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.793577 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-env-overrides\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.794006 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793583 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cni-binary-copy\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.794006 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793623 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-k8s-cni-cncf-io\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.794006 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793828 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-bin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.794006 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793861 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-tmp-dir\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:04.794006 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793863 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-iptables-alerter-script\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:04.794006 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793942 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-systemd-units\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.794006 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.793974 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-etc-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.794345 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794020 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.794345 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794050 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-script-lib\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.794345 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-etc-selinux\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.794345 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794103 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-hostroot\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.794345 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794131 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgf48\" (UniqueName: \"kubernetes.io/projected/288087a8-c7ea-40a8-88bb-1b85be0aef55-kube-api-access-tgf48\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:04.794345 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794234 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmqn\" (UniqueName: \"kubernetes.io/projected/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-kube-api-access-ncmqn\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:04.794345 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794262 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-bin\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.794345 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794288 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-kubernetes\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.794345 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794315 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-conf\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.794345 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794341 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.794836 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794367 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-socket-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.794836 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794709 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-env-overrides\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.794836 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794785 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-etc-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.794836 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794793 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-systemd-units\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.795038 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794849 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-bin\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.795038 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794853 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.795038 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.794882 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-iptables-alerter-script\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:04.795193 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.795140 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa9e011c-a51f-4738-b970-58ec77045b81-ovn-node-metrics-cert\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.795386 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.795364 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-script-lib\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.795717 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.795699 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aaae2c3-0ea2-4061-a448-e50f82dcf823-agent-certs\") pod \"konnectivity-agent-rwtn6\" (UID: \"1aaae2c3-0ea2-4061-a448-e50f82dcf823\") " pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:04.809415 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.809398 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:59:04.809523 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.809420 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:59:04.809523 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.809435 2583 projected.go:196] Error preparing data for projected volume kube-api-access-r8j2z for pod openshift-network-diagnostics/network-check-target-sjsfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:04.809523 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:04.809491 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z podName:09700847-fa74-4444-9d56-5a4e45300c79 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:05.309474818 +0000 UTC m=+2.040088671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r8j2z" (UniqueName: "kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z") pod "network-check-target-sjsfh" (UID: "09700847-fa74-4444-9d56-5a4e45300c79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:04.811086 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.810885 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9hhh\" (UniqueName: \"kubernetes.io/projected/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-kube-api-access-w9hhh\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:04.811086 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.811060 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svj5d\" (UniqueName: \"kubernetes.io/projected/fa9e011c-a51f-4738-b970-58ec77045b81-kube-api-access-svj5d\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:04.812589 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.812567 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmqn\" (UniqueName: \"kubernetes.io/projected/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-kube-api-access-ncmqn\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:04.812697 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.812578 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgf48\" (UniqueName: \"kubernetes.io/projected/288087a8-c7ea-40a8-88bb-1b85be0aef55-kube-api-access-tgf48\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:04.894614 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-host\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.894719 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-etc-tuned\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.894719 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-tmp\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.894719 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894669 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-system-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.894719 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894701 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-system-cni-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.894719 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894706 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-host\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wgc\" (UniqueName: \"kubernetes.io/projected/ef470cac-0cd7-4fae-87e2-f6d431227d00-kube-api-access-d7wgc\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894771 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-system-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mll9t\" (UniqueName: \"kubernetes.io/projected/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kube-api-access-mll9t\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894828 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-lib-modules\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894853 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-multus\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894796 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-system-cni-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894881 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-conf-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894908 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894932 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-run\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894961 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwpx\" (UniqueName: \"kubernetes.io/projected/5d525238-6ced-4f9f-89a6-56fb50fba909-kube-api-access-jtwpx\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:04.894996 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-modprobe-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894977 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-conf-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895015 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d525238-6ced-4f9f-89a6-56fb50fba909-serviceca\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895029 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-multus\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895042 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cnibin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895087 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-multus-certs\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895109 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cnibin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895124 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-device-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895075 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.894986 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-lib-modules\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895113 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-modprobe-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895164 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-run\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895203 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-device-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895208 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-etc-kubernetes\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-daemon-config\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895243 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-multus-certs\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.895583 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895325 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-socket-dir-parent\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895352 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw5pd\" (UniqueName: \"kubernetes.io/projected/7bf98082-9c6b-44e8-99e1-8ad8320b3483-kube-api-access-bw5pd\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-cnibin\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-registration-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-sys-fs\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895451 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysconfig\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895295 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-etc-kubernetes\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895539 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-sys\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895553 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-socket-dir-parent\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-systemd\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895697 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d525238-6ced-4f9f-89a6-56fb50fba909-serviceca\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895756 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysconfig\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895770 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-sys\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895785 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpfl\" (UniqueName: \"kubernetes.io/projected/cabced46-3402-4172-aa1b-f056b9e01220-kube-api-access-xfpfl\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895809 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-kubelet\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895822 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-registration-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.896437 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895851 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cni-binary-copy\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895872 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-k8s-cni-cncf-io\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895885 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-cnibin\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895895 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-bin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895915 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-daemon-config\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895932 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-sys-fs\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.895983 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-bin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896016 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-etc-selinux\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896055 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-k8s-cni-cncf-io\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896077 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-hostroot\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896083 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-kubelet\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896144 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-etc-selinux\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896149 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-hostroot\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896151 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-systemd\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-kubernetes\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-conf\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896313 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-kubernetes\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896356 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.897168 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896380 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-socket-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896438 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896524 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-conf\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896545 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-binary-copy\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896715 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d525238-6ced-4f9f-89a6-56fb50fba909-host\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896744 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-var-lib-kubelet\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896726 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896795 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896825 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d525238-6ced-4f9f-89a6-56fb50fba909-host\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896835 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-netns\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896888 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-var-lib-kubelet\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896908 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.896916 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-netns\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.897795 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.897523 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-socket-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.898905 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.898267 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-etc-tuned\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.898905 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.898338 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-tmp\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.898905 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.898431 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-binary-copy\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.898905 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.898492 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cni-binary-copy\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:04.898905 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.898592 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.901336 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:04.901303 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7793224c37886f13184901ef3d534bb5.slice/crio-256e47eee838717b2a5550e39abbf29ab90699afeb6eb49bd5be65e316afe491 WatchSource:0}: Error finding container 256e47eee838717b2a5550e39abbf29ab90699afeb6eb49bd5be65e316afe491: Status 404 returned error can't find the container with id 256e47eee838717b2a5550e39abbf29ab90699afeb6eb49bd5be65e316afe491 Oct 31 12:59:04.901793 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:04.901774 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9f494cd77ee5cda911dbf4d5fd20aa.slice/crio-7d0156f0bb202ee1e9a13702d1b9662569f6daa6596a371330024765aa21c1d9 WatchSource:0}: Error finding container 7d0156f0bb202ee1e9a13702d1b9662569f6daa6596a371330024765aa21c1d9: Status 404 returned error can't find the container with id 7d0156f0bb202ee1e9a13702d1b9662569f6daa6596a371330024765aa21c1d9 Oct 31 12:59:04.906770 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.906754 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:59:04.909952 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.909881 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpfl\" (UniqueName: \"kubernetes.io/projected/cabced46-3402-4172-aa1b-f056b9e01220-kube-api-access-xfpfl\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:04.909952 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.909917 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mll9t\" (UniqueName: \"kubernetes.io/projected/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kube-api-access-mll9t\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:04.909952 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.909937 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwpx\" (UniqueName: \"kubernetes.io/projected/5d525238-6ced-4f9f-89a6-56fb50fba909-kube-api-access-jtwpx\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:04.910196 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.910173 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wgc\" (UniqueName: \"kubernetes.io/projected/ef470cac-0cd7-4fae-87e2-f6d431227d00-kube-api-access-d7wgc\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:04.910300 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:04.910284 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw5pd\" (UniqueName: \"kubernetes.io/projected/7bf98082-9c6b-44e8-99e1-8ad8320b3483-kube-api-access-bw5pd\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:05.007080 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.007011 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:05.017033 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.017009 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:05.017755 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:05.017731 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aaae2c3_0ea2_4061_a448_e50f82dcf823.slice/crio-4614d6bfe74b19056ef5e1c8e436af77da12792b1c2abdad477e3cf7025a7d60 WatchSource:0}: Error finding container 4614d6bfe74b19056ef5e1c8e436af77da12792b1c2abdad477e3cf7025a7d60: Status 404 returned error can't find the container with id 4614d6bfe74b19056ef5e1c8e436af77da12792b1c2abdad477e3cf7025a7d60 Oct 31 12:59:05.027139 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:05.027117 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd465d2_5a1c_4fd4_b43f_a386fc814e3f.slice/crio-5f9b48e55549ee1f76bb3a9f20a3e727fab4a390369a40f2599574357d79baca WatchSource:0}: Error finding container 5f9b48e55549ee1f76bb3a9f20a3e727fab4a390369a40f2599574357d79baca: Status 404 returned error can't find the container with id 5f9b48e55549ee1f76bb3a9f20a3e727fab4a390369a40f2599574357d79baca Oct 31 12:59:05.064280 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.064257 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:05.064363 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.064313 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:05.069972 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.069954 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:05.074027 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:05.074005 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded98cbbd_063e_43ed_bddc_ba216fbf56bb.slice/crio-6c57f50c7d7ef264584224c6e6077cfaea3f1c5aaa7c7518b5e4f0fe81d94de8 WatchSource:0}: Error finding container 6c57f50c7d7ef264584224c6e6077cfaea3f1c5aaa7c7518b5e4f0fe81d94de8: Status 404 returned error can't find the container with id 6c57f50c7d7ef264584224c6e6077cfaea3f1c5aaa7c7518b5e4f0fe81d94de8 Oct 31 12:59:05.077341 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:05.077319 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9e011c_a51f_4738_b970_58ec77045b81.slice/crio-bb6bfebd88384d0f5871c17765d681316d4abaed0ed6190bcea80caa997a96ef WatchSource:0}: Error finding container bb6bfebd88384d0f5871c17765d681316d4abaed0ed6190bcea80caa997a96ef: Status 404 returned error can't find the container with id bb6bfebd88384d0f5871c17765d681316d4abaed0ed6190bcea80caa997a96ef Oct 31 12:59:05.080481 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:05.080452 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d525238_6ced_4f9f_89a6_56fb50fba909.slice/crio-616e70f1a66a6f876a1ba23e7a895523dc3c488c3dc17c2a36e0d2d5e260fbe5 WatchSource:0}: Error finding container 616e70f1a66a6f876a1ba23e7a895523dc3c488c3dc17c2a36e0d2d5e260fbe5: Status 404 returned error can't find the container with id 616e70f1a66a6f876a1ba23e7a895523dc3c488c3dc17c2a36e0d2d5e260fbe5 Oct 31 12:59:05.091185 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.091167 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:05.100035 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.100013 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:05.101224 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:05.101198 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef470cac_0cd7_4fae_87e2_f6d431227d00.slice/crio-d9fcedf6d05e9ccfdd12ff1235c323d6faa9d63db4729007e0adce5463026ad3 WatchSource:0}: Error finding container d9fcedf6d05e9ccfdd12ff1235c323d6faa9d63db4729007e0adce5463026ad3: Status 404 returned error can't find the container with id d9fcedf6d05e9ccfdd12ff1235c323d6faa9d63db4729007e0adce5463026ad3 Oct 31 12:59:05.103182 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.102963 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:05.108544 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.108526 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vgm78" Oct 31 12:59:05.108789 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:05.108769 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a2f7b9_0ca0_4670_80cd_16e31aecb597.slice/crio-20e9d7a3ff5e237a1dbc6bd4d5b64abf01a99db44b7a25bf2e0835d373536346 WatchSource:0}: Error finding container 20e9d7a3ff5e237a1dbc6bd4d5b64abf01a99db44b7a25bf2e0835d373536346: Status 404 returned error can't find the container with id 20e9d7a3ff5e237a1dbc6bd4d5b64abf01a99db44b7a25bf2e0835d373536346 Oct 31 12:59:05.111314 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:05.111296 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabced46_3402_4172_aa1b_f056b9e01220.slice/crio-40dca8c7dfc0e5b7fc448c1d16d29c67735ab002a4f87c380321ca6bba9e0306 WatchSource:0}: Error finding container 40dca8c7dfc0e5b7fc448c1d16d29c67735ab002a4f87c380321ca6bba9e0306: Status 404 returned error can't find the container with id 40dca8c7dfc0e5b7fc448c1d16d29c67735ab002a4f87c380321ca6bba9e0306 Oct 31 12:59:05.117685 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:05.117661 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf98082_9c6b_44e8_99e1_8ad8320b3483.slice/crio-06eaee4b4a0d86a988b1a795d07f34a7698d5428b6c43e4622134721248570fc WatchSource:0}: Error finding container 06eaee4b4a0d86a988b1a795d07f34a7698d5428b6c43e4622134721248570fc: Status 404 returned error can't find the container with id 06eaee4b4a0d86a988b1a795d07f34a7698d5428b6c43e4622134721248570fc Oct 31 12:59:05.203827 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.203808 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:59:05.237273 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.237254 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:59:05.299811 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.299751 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:05.299811 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.299788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:05.299939 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:05.299906 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:05.299975 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:05.299963 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs podName:288087a8-c7ea-40a8-88bb-1b85be0aef55 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:06.299950782 +0000 UTC m=+3.030564629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs") pod "network-metrics-daemon-sq2q6" (UID: "288087a8-c7ea-40a8-88bb-1b85be0aef55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:05.300018 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:05.299906 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:05.300051 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:05.300035 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret podName:d83f70c7-ab6c-43b0-b86d-3d9d935973ec nodeName:}" failed. No retries permitted until 2026-10-31 12:59:06.30002257 +0000 UTC m=+3.030636428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret") pod "global-pull-secret-syncer-w58sk" (UID: "d83f70c7-ab6c-43b0-b86d-3d9d935973ec") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:05.400197 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.400146 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:05.400377 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:05.400361 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:59:05.400442 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:05.400387 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:59:05.400442 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:05.400401 2583 projected.go:196] Error preparing data for projected volume kube-api-access-r8j2z for pod openshift-network-diagnostics/network-check-target-sjsfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:05.400664 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:05.400460 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z podName:09700847-fa74-4444-9d56-5a4e45300c79 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:06.400442149 +0000 UTC m=+3.131055993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8j2z" (UniqueName: "kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z") pod "network-check-target-sjsfh" (UID: "09700847-fa74-4444-9d56-5a4e45300c79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:05.774043 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.773887 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vgm78" event={"ID":"7bf98082-9c6b-44e8-99e1-8ad8320b3483","Type":"ContainerStarted","Data":"06eaee4b4a0d86a988b1a795d07f34a7698d5428b6c43e4622134721248570fc"} Oct 31 12:59:05.782467 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.782390 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" event={"ID":"cabced46-3402-4172-aa1b-f056b9e01220","Type":"ContainerStarted","Data":"40dca8c7dfc0e5b7fc448c1d16d29c67735ab002a4f87c380321ca6bba9e0306"} Oct 31 12:59:05.784696 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.784609 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:54:05 +0000 UTC" deadline="2028-03-31 09:54:41.510748718 +0000 UTC" Oct 31 12:59:05.784696 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.784674 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12404h55m35.726079029s" Oct 31 12:59:05.798926 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.798897 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s7fjc" event={"ID":"5d525238-6ced-4f9f-89a6-56fb50fba909","Type":"ContainerStarted","Data":"616e70f1a66a6f876a1ba23e7a895523dc3c488c3dc17c2a36e0d2d5e260fbe5"} Oct 31 12:59:05.816016 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.815966 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" event={"ID":"fa9e011c-a51f-4738-b970-58ec77045b81","Type":"ContainerStarted","Data":"bb6bfebd88384d0f5871c17765d681316d4abaed0ed6190bcea80caa997a96ef"} Oct 31 12:59:05.825449 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.825403 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cs9pm" event={"ID":"ed98cbbd-063e-43ed-bddc-ba216fbf56bb","Type":"ContainerStarted","Data":"6c57f50c7d7ef264584224c6e6077cfaea3f1c5aaa7c7518b5e4f0fe81d94de8"} Oct 31 12:59:05.833600 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.833537 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l2brm" event={"ID":"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f","Type":"ContainerStarted","Data":"5f9b48e55549ee1f76bb3a9f20a3e727fab4a390369a40f2599574357d79baca"} Oct 31 12:59:05.840978 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.840949 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rwtn6" event={"ID":"1aaae2c3-0ea2-4061-a448-e50f82dcf823","Type":"ContainerStarted","Data":"4614d6bfe74b19056ef5e1c8e436af77da12792b1c2abdad477e3cf7025a7d60"} Oct 31 12:59:05.845115 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.845085 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" event={"ID":"7793224c37886f13184901ef3d534bb5","Type":"ContainerStarted","Data":"256e47eee838717b2a5550e39abbf29ab90699afeb6eb49bd5be65e316afe491"} Oct 31 12:59:05.849323 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.849294 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" event={"ID":"a5a2f7b9-0ca0-4670-80cd-16e31aecb597","Type":"ContainerStarted","Data":"20e9d7a3ff5e237a1dbc6bd4d5b64abf01a99db44b7a25bf2e0835d373536346"} Oct 31 12:59:05.856453 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.856426 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprss" event={"ID":"ef470cac-0cd7-4fae-87e2-f6d431227d00","Type":"ContainerStarted","Data":"d9fcedf6d05e9ccfdd12ff1235c323d6faa9d63db4729007e0adce5463026ad3"} Oct 31 12:59:05.858978 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:05.858953 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" event={"ID":"df9f494cd77ee5cda911dbf4d5fd20aa","Type":"ContainerStarted","Data":"7d0156f0bb202ee1e9a13702d1b9662569f6daa6596a371330024765aa21c1d9"} Oct 31 12:59:06.137447 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:06.137365 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:59:06.307935 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:06.307897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:06.308114 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:06.308007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:06.308206 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.308187 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:06.308272 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.308260 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret podName:d83f70c7-ab6c-43b0-b86d-3d9d935973ec nodeName:}" failed. No retries permitted until 2026-10-31 12:59:08.308240424 +0000 UTC m=+5.038854273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret") pod "global-pull-secret-syncer-w58sk" (UID: "d83f70c7-ab6c-43b0-b86d-3d9d935973ec") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:06.308752 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.308729 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:06.308874 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.308822 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs podName:288087a8-c7ea-40a8-88bb-1b85be0aef55 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:08.3088064 +0000 UTC m=+5.039420244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs") pod "network-metrics-daemon-sq2q6" (UID: "288087a8-c7ea-40a8-88bb-1b85be0aef55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:06.410598 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:06.409804 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:06.410598 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.410060 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:59:06.410598 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.410094 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:59:06.410598 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.410110 2583 projected.go:196] Error preparing data for projected volume kube-api-access-r8j2z for pod openshift-network-diagnostics/network-check-target-sjsfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:06.410598 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.410213 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z podName:09700847-fa74-4444-9d56-5a4e45300c79 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:08.410173365 +0000 UTC m=+5.140787234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8j2z" (UniqueName: "kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z") pod "network-check-target-sjsfh" (UID: "09700847-fa74-4444-9d56-5a4e45300c79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:06.756096 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:06.756013 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:06.756262 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.756158 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:06.756623 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:06.756597 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:06.756723 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.756696 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:06.756787 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:06.756770 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:06.756920 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:06.756885 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:06.785247 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:06.785185 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:54:05 +0000 UTC" deadline="2028-06-03 15:03:33.749993176 +0000 UTC" Oct 31 12:59:06.785247 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:06.785244 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13946h4m26.964753539s" Oct 31 12:59:08.330093 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:08.329232 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:08.330093 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:08.329291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:08.330093 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.329489 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:08.330093 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.329565 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret podName:d83f70c7-ab6c-43b0-b86d-3d9d935973ec nodeName:}" failed. No retries permitted until 2026-10-31 12:59:12.329546541 +0000 UTC m=+9.060160400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret") pod "global-pull-secret-syncer-w58sk" (UID: "d83f70c7-ab6c-43b0-b86d-3d9d935973ec") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:08.330093 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.330002 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:08.330093 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.330056 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs podName:288087a8-c7ea-40a8-88bb-1b85be0aef55 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:12.330039211 +0000 UTC m=+9.060653071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs") pod "network-metrics-daemon-sq2q6" (UID: "288087a8-c7ea-40a8-88bb-1b85be0aef55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:08.431096 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:08.430417 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:08.431096 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.430608 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:59:08.431096 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.430628 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:59:08.431096 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.430639 2583 projected.go:196] Error preparing data for projected volume kube-api-access-r8j2z for pod openshift-network-diagnostics/network-check-target-sjsfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:08.431096 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.430704 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z podName:09700847-fa74-4444-9d56-5a4e45300c79 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:12.430686188 +0000 UTC m=+9.161300042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8j2z" (UniqueName: "kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z") pod "network-check-target-sjsfh" (UID: "09700847-fa74-4444-9d56-5a4e45300c79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:08.756316 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:08.756283 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:08.756494 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.756405 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:08.756572 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:08.756535 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:08.756655 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.756634 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:08.756989 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:08.756468 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:08.757093 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:08.757071 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:09.899833 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:09.898841 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rwtn6" event={"ID":"1aaae2c3-0ea2-4061-a448-e50f82dcf823","Type":"ContainerStarted","Data":"d6cf66b03763417f3aa173d0b513cb07d9868d281f49c5e88419aeed6796e20e"} Oct 31 12:59:09.905114 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:09.904487 2583 generic.go:356] "Generic (PLEG): container finished" podID="7793224c37886f13184901ef3d534bb5" containerID="23b8079f1f1ece8c5d3772d16ed9b051482f80bf291fdc24b53e0d1413b40d0e" exitCode=0 Oct 31 12:59:09.905114 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:09.904572 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" event={"ID":"7793224c37886f13184901ef3d534bb5","Type":"ContainerDied","Data":"23b8079f1f1ece8c5d3772d16ed9b051482f80bf291fdc24b53e0d1413b40d0e"} Oct 31 12:59:09.908535 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:09.907779 2583 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="0dd31abbf2a717c26a8854ee045466afdffc0113bd7ecac150f57f6703fe3d3d" exitCode=0 Oct 31 12:59:09.908535 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:09.907849 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprss" event={"ID":"ef470cac-0cd7-4fae-87e2-f6d431227d00","Type":"ContainerDied","Data":"0dd31abbf2a717c26a8854ee045466afdffc0113bd7ecac150f57f6703fe3d3d"} Oct 31 12:59:09.915704 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:09.915682 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" event={"ID":"df9f494cd77ee5cda911dbf4d5fd20aa","Type":"ContainerStarted","Data":"17e8aefe897f0ca8e115165c8a997434823ceb2c4dfe6a26f287f4fa359134a1"} Oct 31 12:59:09.945962 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:09.944876 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rwtn6" podStartSLOduration=1.418970418 podStartE2EDuration="5.944859594s" totalImagesPullingTime="4.525889176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:09.918696826 +0000 UTC m=+6.649310688" watchObservedRunningTime="2026-10-31 12:59:09.944859594 +0000 UTC m=+6.675473454" Oct 31 12:59:10.756045 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:10.756010 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:10.756212 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:10.756148 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:10.756647 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:10.756626 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:10.756764 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:10.756743 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:10.756818 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:10.756802 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:10.756938 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:10.756922 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:10.923422 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:10.923385 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" event={"ID":"7793224c37886f13184901ef3d534bb5","Type":"ContainerStarted","Data":"52ff25f9d1c11dcda28035fd35359ae8b116797c9d58cd6eaceeebb088c4279b"} Oct 31 12:59:10.941196 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:10.941119 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" podStartSLOduration=6.941101871 podStartE2EDuration="6.941101871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:09.981789318 +0000 UTC m=+6.712403181" watchObservedRunningTime="2026-10-31 12:59:10.941101871 +0000 UTC m=+7.671715734" Oct 31 12:59:11.929772 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:11.929050 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s7fjc" event={"ID":"5d525238-6ced-4f9f-89a6-56fb50fba909","Type":"ContainerStarted","Data":"9be90f1d8e489f7d219271148e6a11b2ddc29426e6bb0f2551cd4f73a77ef89f"} Oct 31 12:59:11.933508 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:11.933178 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" event={"ID":"a5a2f7b9-0ca0-4670-80cd-16e31aecb597","Type":"ContainerStarted","Data":"37f04fcc8f8fd7cb73b962a202620016e921f22aed74a0a5398537bdf15ba785"} Oct 31 12:59:11.948219 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:11.948174 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" podStartSLOduration=7.948162282 podStartE2EDuration="7.948162282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:10.941825687 +0000 UTC m=+7.672439557" watchObservedRunningTime="2026-10-31 12:59:11.948162282 +0000 UTC m=+8.678776143" Oct 31 12:59:12.366572 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:12.365783 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:12.366572 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:12.365836 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:12.366572 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.366015 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:12.366572 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.366075 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret podName:d83f70c7-ab6c-43b0-b86d-3d9d935973ec nodeName:}" failed. No retries permitted until 2026-10-31 12:59:20.366057914 +0000 UTC m=+17.096671778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret") pod "global-pull-secret-syncer-w58sk" (UID: "d83f70c7-ab6c-43b0-b86d-3d9d935973ec") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:12.366572 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.366464 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:12.366572 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.366531 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs podName:288087a8-c7ea-40a8-88bb-1b85be0aef55 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:20.366515782 +0000 UTC m=+17.097129642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs") pod "network-metrics-daemon-sq2q6" (UID: "288087a8-c7ea-40a8-88bb-1b85be0aef55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:12.466786 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:12.466198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:12.466786 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.466356 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:59:12.466786 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.466376 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:59:12.466786 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.466387 2583 projected.go:196] Error preparing data for projected volume kube-api-access-r8j2z for pod openshift-network-diagnostics/network-check-target-sjsfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:12.466786 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.466442 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z podName:09700847-fa74-4444-9d56-5a4e45300c79 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:20.466425891 +0000 UTC m=+17.197039736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8j2z" (UniqueName: "kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z") pod "network-check-target-sjsfh" (UID: "09700847-fa74-4444-9d56-5a4e45300c79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:12.756806 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:12.756007 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:12.756806 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.756127 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:12.756806 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:12.756203 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:12.756806 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.756283 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:12.756806 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:12.756336 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:12.756806 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:12.756408 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:13.308249 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:13.308226 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 12:59:13.766962 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:13.766851 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:59:13.308246002Z","UUID":"8636d13e-a8fe-45a3-9188-31c0f26b95c3","Handler":null,"Name":"","Endpoint":""} Oct 31 12:59:13.768386 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:13.768363 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 12:59:13.768511 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:13.768407 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 12:59:13.942771 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:13.941887 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vgm78" event={"ID":"7bf98082-9c6b-44e8-99e1-8ad8320b3483","Type":"ContainerStarted","Data":"47c1d0f721e7e09b662355e89bd50a8b03c8e9728b7f4d83ccdc42976bb9bcf9"} Oct 31 12:59:13.945713 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:13.945439 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" event={"ID":"a5a2f7b9-0ca0-4670-80cd-16e31aecb597","Type":"ContainerStarted","Data":"b8cbcff08ea5c05399f6549e962e73483c52a7af98d2647f28369c60c5e4a4c0"} Oct 31 12:59:13.965015 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:13.964967 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s7fjc" podStartSLOduration=4.183293847 podStartE2EDuration="9.964950444s" totalImagesPullingTime="5.781656597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:11.948449003 +0000 UTC m=+8.679062865" watchObservedRunningTime="2026-10-31 12:59:13.964950444 +0000 UTC m=+10.695564309" Oct 31 12:59:14.210051 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.209810 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:14.210917 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.210705 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:14.231412 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.230786 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vgm78" podStartSLOduration=2.022916009 podStartE2EDuration="10.230766439s" totalImagesPullingTime="8.20785043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:13.964371763 +0000 UTC m=+10.694985626" watchObservedRunningTime="2026-10-31 12:59:14.230766439 +0000 UTC m=+10.961380314" Oct 31 12:59:14.755756 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.755676 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:14.756300 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:14.755817 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:14.756300 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.756177 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:14.756300 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:14.756257 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:14.756463 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.756323 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:14.756463 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:14.756380 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:14.949872 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.949834 2583 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="72e2d4871ba42c56a68b202fdfbf6e51f119b004f3c7311fa1c397c87fe306fe" exitCode=0 Oct 31 12:59:14.950058 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.949920 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprss" event={"ID":"ef470cac-0cd7-4fae-87e2-f6d431227d00","Type":"ContainerDied","Data":"72e2d4871ba42c56a68b202fdfbf6e51f119b004f3c7311fa1c397c87fe306fe"} Oct 31 12:59:14.951971 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.951941 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cs9pm" event={"ID":"ed98cbbd-063e-43ed-bddc-ba216fbf56bb","Type":"ContainerStarted","Data":"49a0ad25f0b87745db053c200e9c40bb0da1f1f8c40a8966fcd28aa2aede1150"} Oct 31 12:59:14.959581 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.959549 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" event={"ID":"a5a2f7b9-0ca0-4670-80cd-16e31aecb597","Type":"ContainerStarted","Data":"f43b71062956de7d3106f083f302f34488b71aa2a740793698b475744c22ed6d"} Oct 31 12:59:14.992750 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:14.992697 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cs9pm" podStartSLOduration=2.303694041 podStartE2EDuration="10.992682865s" totalImagesPullingTime="8.688988824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:14.992243457 +0000 UTC m=+11.722857319" watchObservedRunningTime="2026-10-31 12:59:14.992682865 +0000 UTC m=+11.723296726" Oct 31 12:59:15.013377 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:15.013271 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" podStartSLOduration=2.163787884 podStartE2EDuration="11.013253308s" totalImagesPullingTime="8.849465424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:15.013029815 +0000 UTC m=+11.743643677" watchObservedRunningTime="2026-10-31 12:59:15.013253308 +0000 UTC m=+11.743867170" Oct 31 12:59:15.962400 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:15.962342 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l2brm" event={"ID":"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f","Type":"ContainerStarted","Data":"88fc79bda1cafa6c03671a1e3975fe88c266b2eab4d83b56bbbf62bb70944bbd"} Oct 31 12:59:15.980281 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:15.980235 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l2brm" podStartSLOduration=3.552674778 podStartE2EDuration="11.980220976s" totalImagesPullingTime="8.427546198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:15.980188449 +0000 UTC m=+12.710802312" watchObservedRunningTime="2026-10-31 12:59:15.980220976 +0000 UTC m=+12.710834839" Oct 31 12:59:16.756009 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:16.755971 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:16.756009 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:16.755999 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:16.756213 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:16.756088 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:16.756213 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:16.756107 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:16.756213 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:16.756199 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:16.756337 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:16.756288 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:16.966441 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:16.966405 2583 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="903c8b7dc27e083a9aa00ca86f85226764d608e1936688b0e06585e9f5d04f38" exitCode=0 Oct 31 12:59:16.967039 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:16.966490 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprss" event={"ID":"ef470cac-0cd7-4fae-87e2-f6d431227d00","Type":"ContainerDied","Data":"903c8b7dc27e083a9aa00ca86f85226764d608e1936688b0e06585e9f5d04f38"} Oct 31 12:59:16.968928 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:16.968827 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" event={"ID":"cabced46-3402-4172-aa1b-f056b9e01220","Type":"ContainerStarted","Data":"f103466d6e5fd523d1daa5a02eb8342c850cf8f6faf0898d25afc1c5e6445e1d"} Oct 31 12:59:17.015304 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:17.015228 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" podStartSLOduration=2.192534299 podStartE2EDuration="13.015215969s" totalImagesPullingTime="10.82268167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:17.014644419 +0000 UTC m=+13.745258282" watchObservedRunningTime="2026-10-31 12:59:17.015215969 +0000 UTC m=+13.745829827" Oct 31 12:59:18.755883 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:18.755691 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:18.756362 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:18.755787 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:18.756362 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:18.755977 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:18.756362 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:18.756067 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:18.756362 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:18.755794 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:18.756362 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:18.756167 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:18.975527 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:18.975493 2583 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="e4e6d47662b7c502b5924c9874ba18d9d5c0ff53b9f8e2d7d42b5f779256ac48" exitCode=0 Oct 31 12:59:18.975670 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:18.975548 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprss" event={"ID":"ef470cac-0cd7-4fae-87e2-f6d431227d00","Type":"ContainerDied","Data":"e4e6d47662b7c502b5924c9874ba18d9d5c0ff53b9f8e2d7d42b5f779256ac48"} Oct 31 12:59:20.423333 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.423164 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:20.423747 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.423361 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:20.423747 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.423317 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:20.423747 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.423463 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs podName:288087a8-c7ea-40a8-88bb-1b85be0aef55 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:36.423444182 +0000 UTC m=+33.154058029 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs") pod "network-metrics-daemon-sq2q6" (UID: "288087a8-c7ea-40a8-88bb-1b85be0aef55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:59:20.423747 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.423509 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:20.423747 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.423553 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret podName:d83f70c7-ab6c-43b0-b86d-3d9d935973ec nodeName:}" failed. No retries permitted until 2026-10-31 12:59:36.423541694 +0000 UTC m=+33.154155534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret") pod "global-pull-secret-syncer-w58sk" (UID: "d83f70c7-ab6c-43b0-b86d-3d9d935973ec") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:59:20.524055 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.523972 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:20.524246 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.524191 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:59:20.524246 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.524219 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:59:20.524246 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.524233 2583 projected.go:196] Error preparing data for projected volume kube-api-access-r8j2z for pod openshift-network-diagnostics/network-check-target-sjsfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:20.524393 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.524303 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z podName:09700847-fa74-4444-9d56-5a4e45300c79 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:36.52428318 +0000 UTC m=+33.254897039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8j2z" (UniqueName: "kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z") pod "network-check-target-sjsfh" (UID: "09700847-fa74-4444-9d56-5a4e45300c79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:59:20.613336 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.613299 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:20.613557 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.613537 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:59:20.614317 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.614125 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:20.755573 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.755551 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:20.755669 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.755577 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:20.755669 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.755645 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:20.755789 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.755711 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:20.755789 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.755768 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:20.755889 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:20.755861 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:20.981778 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.981752 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 12:59:20.982129 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.982110 2583 generic.go:356] "Generic (PLEG): container finished" podID="fa9e011c-a51f-4738-b970-58ec77045b81" containerID="f6d377953c54ad445e341752d13ced406b97de99072a4304b6ab6f0f23f33263" exitCode=1 Oct 31 12:59:20.982207 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.982187 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" event={"ID":"fa9e011c-a51f-4738-b970-58ec77045b81","Type":"ContainerStarted","Data":"fc7febd7a139c1dbe8c75760f5f3d6b54b012f231cede557fa689ceb7d9ae157"} Oct 31 12:59:20.982262 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.982256 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" event={"ID":"fa9e011c-a51f-4738-b970-58ec77045b81","Type":"ContainerStarted","Data":"a1936ac244f69743c4ff09fd8d943b2ec15e017fd21cdb9272d56f4fe08bbb08"} Oct 31 12:59:20.982322 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.982291 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" event={"ID":"fa9e011c-a51f-4738-b970-58ec77045b81","Type":"ContainerStarted","Data":"752ad1be1bd029f31880dff1621bc6664de708c1e89e22ae83e18722b42648cc"} Oct 31 12:59:20.982374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.982326 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" event={"ID":"fa9e011c-a51f-4738-b970-58ec77045b81","Type":"ContainerStarted","Data":"d79011000de551345140c00fca00ecea2c5e2c03c85ad15b694c63d4c1644a24"} Oct 31 12:59:20.982374 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.982358 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" event={"ID":"fa9e011c-a51f-4738-b970-58ec77045b81","Type":"ContainerDied","Data":"f6d377953c54ad445e341752d13ced406b97de99072a4304b6ab6f0f23f33263"} Oct 31 12:59:20.982441 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.982387 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" event={"ID":"fa9e011c-a51f-4738-b970-58ec77045b81","Type":"ContainerStarted","Data":"586a4ebf67e20b3a96c12d28b7af4c981a54a107df6b115868b78b5490f6ca42"} Oct 31 12:59:20.985359 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.985336 2583 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="4971798d01053b7f8621e22f6854e33973269734fda0fc08f42042850c46c364" exitCode=0 Oct 31 12:59:20.985467 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:20.985418 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprss" event={"ID":"ef470cac-0cd7-4fae-87e2-f6d431227d00","Type":"ContainerDied","Data":"4971798d01053b7f8621e22f6854e33973269734fda0fc08f42042850c46c364"} Oct 31 12:59:21.990711 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:21.990675 2583 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="e4d6529470f55ac7954c3ce55cb53bccf1bcb35a82e4afc59420ee02c9b15195" exitCode=0 Oct 31 12:59:21.991241 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:21.990732 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprss" event={"ID":"ef470cac-0cd7-4fae-87e2-f6d431227d00","Type":"ContainerDied","Data":"e4d6529470f55ac7954c3ce55cb53bccf1bcb35a82e4afc59420ee02c9b15195"} Oct 31 12:59:22.755672 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:22.755603 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:22.755781 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:22.755616 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:22.755781 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:22.755696 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:22.755781 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:22.755697 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:22.755781 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:22.755772 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:22.755900 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:22.755857 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:22.996037 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:22.996008 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 12:59:22.996560 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:22.996429 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" event={"ID":"fa9e011c-a51f-4738-b970-58ec77045b81","Type":"ContainerStarted","Data":"27a34bbb78c0441a6c7c28580ac7d2e17c1b544cc8e0997a01e3f362e6590b34"} Oct 31 12:59:22.999590 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:22.999567 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprss" event={"ID":"ef470cac-0cd7-4fae-87e2-f6d431227d00","Type":"ContainerStarted","Data":"087ac845e4a02c369e0e26781bb5e480c734ba0794208b9870fd3e29c46f1e62"} Oct 31 12:59:23.027595 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:23.027516 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wprss" podStartSLOduration=8.046492385 podStartE2EDuration="19.02750107s" totalImagesPullingTime="10.981008685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:23.027326779 +0000 UTC m=+19.757940640" watchObservedRunningTime="2026-10-31 12:59:23.02750107 +0000 UTC m=+19.758114930" Oct 31 12:59:24.755907 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:24.755731 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:24.756397 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:24.755755 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:24.756397 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:24.755976 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:24.756397 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:24.756091 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:24.756397 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:24.755795 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:24.756397 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:24.756199 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:25.006993 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:25.006965 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 12:59:25.007307 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:25.007284 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" event={"ID":"fa9e011c-a51f-4738-b970-58ec77045b81","Type":"ContainerStarted","Data":"9ed8384ced434a13278c71dc35a64c10c926c73a42a656cd6ef0013549d44499"} Oct 31 12:59:25.007607 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:25.007588 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:25.007701 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:25.007650 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:25.007701 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:25.007689 2583 scope.go:122] "RemoveContainer" containerID="f6d377953c54ad445e341752d13ced406b97de99072a4304b6ab6f0f23f33263" Oct 31 12:59:25.024254 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:25.024235 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:26.014382 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.014352 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 12:59:26.014878 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.014814 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" event={"ID":"fa9e011c-a51f-4738-b970-58ec77045b81","Type":"ContainerStarted","Data":"2eae5e4f9aa6827d55e7bd34ec7c7f3a66e6a593ffa8756757b5ce7a0a2c11af"} Oct 31 12:59:26.015195 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.015168 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:26.028911 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.028886 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:26.052775 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.052722 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" podStartSLOduration=7.151817261 podStartE2EDuration="22.052709706s" totalImagesPullingTime="14.900892445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:26.052534621 +0000 UTC m=+22.783148517" watchObservedRunningTime="2026-10-31 12:59:26.052709706 +0000 UTC m=+22.783323569" Oct 31 12:59:26.322503 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.322440 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w58sk"] Oct 31 12:59:26.322641 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.322584 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:26.322711 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:26.322691 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:26.323224 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.323206 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sjsfh"] Oct 31 12:59:26.323303 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.323293 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:26.323383 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:26.323365 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:26.324852 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.324826 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sq2q6"] Oct 31 12:59:26.324969 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:26.324929 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:26.325022 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:26.325009 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:27.756439 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:27.756411 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:27.756932 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:27.756411 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:27.756932 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:27.756706 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:27.756932 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:27.756908 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:27.759854 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:27.757262 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:27.759854 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:27.757503 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:29.756372 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:29.756340 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:29.756372 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:29.756371 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:29.756874 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:29.756429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:29.756874 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:29.756441 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:29.756874 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:29.756558 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:29.756874 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:29.756646 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:31.756034 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:31.755861 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:31.756441 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:31.755869 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:31.756441 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:31.756152 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sq2q6" podUID="288087a8-c7ea-40a8-88bb-1b85be0aef55" Oct 31 12:59:31.756441 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:31.755923 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:31.756441 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:31.756191 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w58sk" podUID="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" Oct 31 12:59:31.756441 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:31.756259 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sjsfh" podUID="09700847-fa74-4444-9d56-5a4e45300c79" Oct 31 12:59:32.037955 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.037884 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeReady" Oct 31 12:59:32.038089 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.037999 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 12:59:32.093028 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.092999 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ltf7k"] Oct 31 12:59:32.096604 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.096588 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rm7b9"] Oct 31 12:59:32.096714 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.096699 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.100339 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.100321 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:32.106404 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.106381 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gth9v\"" Oct 31 12:59:32.106515 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.106385 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 12:59:32.106515 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.106425 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 12:59:32.106515 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.106469 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 12:59:32.106659 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.106609 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 12:59:32.107466 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.107444 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftfm5\"" Oct 31 12:59:32.107581 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.107521 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 12:59:32.108032 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.107979 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rm7b9"] Oct 31 12:59:32.108632 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.108609 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ltf7k"] Oct 31 12:59:32.209300 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.209258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-config-volume\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.209300 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.209295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.209554 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.209318 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872ln\" (UniqueName: \"kubernetes.io/projected/9e16e028-6e02-48e1-b54e-3517232c2d8b-kube-api-access-872ln\") pod \"ingress-canary-rm7b9\" (UID: \"9e16e028-6e02-48e1-b54e-3517232c2d8b\") " pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:32.209554 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.209443 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-tmp-dir\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.209554 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.209462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e16e028-6e02-48e1-b54e-3517232c2d8b-cert\") pod \"ingress-canary-rm7b9\" (UID: \"9e16e028-6e02-48e1-b54e-3517232c2d8b\") " pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:32.209554 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.209493 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65pj\" (UniqueName: \"kubernetes.io/projected/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-kube-api-access-x65pj\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.310701 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.310632 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-config-volume\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.310701 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.310670 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.310701 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.310698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872ln\" (UniqueName: \"kubernetes.io/projected/9e16e028-6e02-48e1-b54e-3517232c2d8b-kube-api-access-872ln\") pod \"ingress-canary-rm7b9\" (UID: \"9e16e028-6e02-48e1-b54e-3517232c2d8b\") " pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:32.310885 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.310750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-tmp-dir\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.310885 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.310771 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e16e028-6e02-48e1-b54e-3517232c2d8b-cert\") pod \"ingress-canary-rm7b9\" (UID: \"9e16e028-6e02-48e1-b54e-3517232c2d8b\") " pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:32.310885 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.310790 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65pj\" (UniqueName: \"kubernetes.io/projected/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-kube-api-access-x65pj\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.310885 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:32.310799 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:59:32.310885 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:32.310871 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls podName:f338bc5c-18e1-4e62-be2d-93de38d4ae9d nodeName:}" failed. No retries permitted until 2026-10-31 12:59:32.810855333 +0000 UTC m=+29.541469186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls") pod "dns-default-ltf7k" (UID: "f338bc5c-18e1-4e62-be2d-93de38d4ae9d") : secret "dns-default-metrics-tls" not found Oct 31 12:59:32.311132 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.311109 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-tmp-dir\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.311498 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.311457 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-config-volume\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.314719 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.314702 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e16e028-6e02-48e1-b54e-3517232c2d8b-cert\") pod \"ingress-canary-rm7b9\" (UID: \"9e16e028-6e02-48e1-b54e-3517232c2d8b\") " pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:32.324286 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.324254 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872ln\" (UniqueName: \"kubernetes.io/projected/9e16e028-6e02-48e1-b54e-3517232c2d8b-kube-api-access-872ln\") pod \"ingress-canary-rm7b9\" (UID: \"9e16e028-6e02-48e1-b54e-3517232c2d8b\") " pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:32.324372 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.324259 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65pj\" (UniqueName: \"kubernetes.io/projected/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-kube-api-access-x65pj\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.417572 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.417548 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:32.705743 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.705713 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rm7b9"] Oct 31 12:59:32.729050 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:32.729027 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e16e028_6e02_48e1_b54e_3517232c2d8b.slice/crio-d4fb16a264a1a0b4ad825bf196dceaecf7ec96056a69bca47f4a6d8658bad448 WatchSource:0}: Error finding container d4fb16a264a1a0b4ad825bf196dceaecf7ec96056a69bca47f4a6d8658bad448: Status 404 returned error can't find the container with id d4fb16a264a1a0b4ad825bf196dceaecf7ec96056a69bca47f4a6d8658bad448 Oct 31 12:59:32.813909 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:32.813884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:32.814414 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:32.813991 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:59:32.814414 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:32.814037 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls podName:f338bc5c-18e1-4e62-be2d-93de38d4ae9d nodeName:}" failed. No retries permitted until 2026-10-31 12:59:33.814022345 +0000 UTC m=+30.544636205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls") pod "dns-default-ltf7k" (UID: "f338bc5c-18e1-4e62-be2d-93de38d4ae9d") : secret "dns-default-metrics-tls" not found Oct 31 12:59:33.031285 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.031199 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rm7b9" event={"ID":"9e16e028-6e02-48e1-b54e-3517232c2d8b","Type":"ContainerStarted","Data":"d4fb16a264a1a0b4ad825bf196dceaecf7ec96056a69bca47f4a6d8658bad448"} Oct 31 12:59:33.448780 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.448752 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-q8jf6-vd797"] Oct 31 12:59:33.456132 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.456110 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c2knk"] Oct 31 12:59:33.456263 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.456247 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.458803 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.458784 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.463607 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.463583 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q8jf6-vd797"] Oct 31 12:59:33.467701 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.467683 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lt77s\"" Oct 31 12:59:33.467799 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.467691 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 12:59:33.467799 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.467727 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 12:59:33.467937 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.467808 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 12:59:33.468105 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.468089 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2knk"] Oct 31 12:59:33.472160 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.472142 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 12:59:33.472219 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.472204 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 12:59:33.472255 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.472234 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qpv6m\"" Oct 31 12:59:33.476419 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.476398 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 12:59:33.518073 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.518051 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.518170 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.518080 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-trusted-ca-bundle\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.518170 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.518116 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.518290 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.518268 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6266aa7-b443-49e9-88a0-987228617831-archives-path\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.518334 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.518302 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-service-ca-bundle\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.518334 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.518318 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4fm9\" (UniqueName: \"kubernetes.io/projected/c6266aa7-b443-49e9-88a0-987228617831-kube-api-access-z4fm9\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.518405 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.518354 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-data-volume\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.518405 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.518386 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-crio-socket\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.518469 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.518425 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnj2\" (UniqueName: \"kubernetes.io/projected/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-api-access-xjnj2\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.618875 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.618836 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-service-ca-bundle\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.618875 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.618877 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4fm9\" (UniqueName: \"kubernetes.io/projected/c6266aa7-b443-49e9-88a0-987228617831-kube-api-access-z4fm9\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.619117 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.618897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-data-volume\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.619117 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.618915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-crio-socket\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.619117 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.619026 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnj2\" (UniqueName: \"kubernetes.io/projected/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-api-access-xjnj2\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.619117 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.619086 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-crio-socket\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.619117 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.619090 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.619117 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.619115 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-trusted-ca-bundle\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.619299 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.619158 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.619299 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.619218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6266aa7-b443-49e9-88a0-987228617831-archives-path\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.619299 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.619220 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-data-volume\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.619521 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:33.619503 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 12:59:33.619572 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.619505 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6266aa7-b443-49e9-88a0-987228617831-archives-path\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.619572 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:33.619565 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:59:33.619669 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.619569 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-service-ca-bundle\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.619669 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:33.619580 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm podName:7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:34.119560901 +0000 UTC m=+30.850174768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2knk" (UID: "7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400") : configmap "kube-rbac-proxy" not found Oct 31 12:59:33.619669 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:33.619631 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls podName:7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:34.119612657 +0000 UTC m=+30.850226500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2knk" (UID: "7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400") : secret "insights-runtime-extractor-tls" not found Oct 31 12:59:33.620109 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.620084 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-trusted-ca-bundle\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.636075 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.636054 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnj2\" (UniqueName: \"kubernetes.io/projected/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-api-access-xjnj2\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:33.636155 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.636093 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4fm9\" (UniqueName: \"kubernetes.io/projected/c6266aa7-b443-49e9-88a0-987228617831-kube-api-access-z4fm9\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.756847 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.756779 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:33.756986 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.756919 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:33.756986 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.756945 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:33.763058 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.763038 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:59:33.763179 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.763094 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:59:33.763179 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.763098 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g22bl\"" Oct 31 12:59:33.763179 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.763131 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:59:33.763179 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.763155 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lsjfn\"" Oct 31 12:59:33.763351 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.763269 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lt4jt\"" Oct 31 12:59:33.763351 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.763273 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:59:33.773512 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.773494 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:33.820439 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.820419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:33.820717 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:33.820550 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:59:33.820717 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:33.820597 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls podName:f338bc5c-18e1-4e62-be2d-93de38d4ae9d nodeName:}" failed. No retries permitted until 2026-10-31 12:59:35.820584684 +0000 UTC m=+32.551198525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls") pod "dns-default-ltf7k" (UID: "f338bc5c-18e1-4e62-be2d-93de38d4ae9d") : secret "dns-default-metrics-tls" not found Oct 31 12:59:33.872900 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.872868 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr"] Oct 31 12:59:33.887140 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.887115 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr" Oct 31 12:59:33.894560 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.894516 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr"] Oct 31 12:59:33.894698 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.894676 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 12:59:33.894782 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.894725 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lmj6s\"" Oct 31 12:59:33.894970 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:33.894954 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 12:59:34.022417 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:34.022336 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plv7c\" (UniqueName: \"kubernetes.io/projected/116df32b-c0fe-456f-bea6-2fdf33dc0039-kube-api-access-plv7c\") pod \"migrator-66df5d496b-lt9fr\" (UID: \"116df32b-c0fe-456f-bea6-2fdf33dc0039\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr" Oct 31 12:59:34.071195 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:34.071167 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q8jf6-vd797"] Oct 31 12:59:34.075770 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:34.075733 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6266aa7_b443_49e9_88a0_987228617831.slice/crio-d79bd8837dd89a25bb7e97a9c72976dc77173457e3393580df8cfadd3e540867 WatchSource:0}: Error finding container d79bd8837dd89a25bb7e97a9c72976dc77173457e3393580df8cfadd3e540867: Status 404 returned error can't find the container with id d79bd8837dd89a25bb7e97a9c72976dc77173457e3393580df8cfadd3e540867 Oct 31 12:59:34.123439 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:34.123411 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:34.123559 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:34.123461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plv7c\" (UniqueName: \"kubernetes.io/projected/116df32b-c0fe-456f-bea6-2fdf33dc0039-kube-api-access-plv7c\") pod \"migrator-66df5d496b-lt9fr\" (UID: \"116df32b-c0fe-456f-bea6-2fdf33dc0039\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr" Oct 31 12:59:34.123559 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:34.123507 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:34.123559 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:34.123555 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:59:34.123710 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:34.123563 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 12:59:34.123710 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:34.123602 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls podName:7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:35.123590132 +0000 UTC m=+31.854203985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2knk" (UID: "7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400") : secret "insights-runtime-extractor-tls" not found Oct 31 12:59:34.123710 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:34.123617 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm podName:7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:35.123610024 +0000 UTC m=+31.854223864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2knk" (UID: "7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400") : configmap "kube-rbac-proxy" not found Oct 31 12:59:34.136794 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:34.136775 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plv7c\" (UniqueName: \"kubernetes.io/projected/116df32b-c0fe-456f-bea6-2fdf33dc0039-kube-api-access-plv7c\") pod \"migrator-66df5d496b-lt9fr\" (UID: \"116df32b-c0fe-456f-bea6-2fdf33dc0039\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr" Oct 31 12:59:34.212870 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:34.212846 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr" Oct 31 12:59:34.509545 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:34.509434 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr"] Oct 31 12:59:34.512324 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:34.512290 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116df32b_c0fe_456f_bea6_2fdf33dc0039.slice/crio-6ba7ad2e2d6a4d8240b1441318185d91836bd5fc221bf0d71130a7dc20ef5b7a WatchSource:0}: Error finding container 6ba7ad2e2d6a4d8240b1441318185d91836bd5fc221bf0d71130a7dc20ef5b7a: Status 404 returned error can't find the container with id 6ba7ad2e2d6a4d8240b1441318185d91836bd5fc221bf0d71130a7dc20ef5b7a Oct 31 12:59:35.037337 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.037301 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr" event={"ID":"116df32b-c0fe-456f-bea6-2fdf33dc0039","Type":"ContainerStarted","Data":"6ba7ad2e2d6a4d8240b1441318185d91836bd5fc221bf0d71130a7dc20ef5b7a"} Oct 31 12:59:35.038369 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.038348 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8jf6-vd797" event={"ID":"c6266aa7-b443-49e9-88a0-987228617831","Type":"ContainerStarted","Data":"d79bd8837dd89a25bb7e97a9c72976dc77173457e3393580df8cfadd3e540867"} Oct 31 12:59:35.132539 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.132505 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:35.132675 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.132578 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:35.132727 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:35.132685 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 12:59:35.132758 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:35.132745 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm podName:7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:37.132728271 +0000 UTC m=+33.863342125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2knk" (UID: "7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400") : configmap "kube-rbac-proxy" not found Oct 31 12:59:35.132812 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:35.132685 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:59:35.132863 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:35.132834 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls podName:7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:37.132813468 +0000 UTC m=+33.863427340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2knk" (UID: "7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400") : secret "insights-runtime-extractor-tls" not found Oct 31 12:59:35.745590 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.745555 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6c5t9"] Oct 31 12:59:35.775306 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.775194 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:35.779990 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.779956 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6c5t9"] Oct 31 12:59:35.781335 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.780960 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 12:59:35.781335 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.781059 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jl9zz\"" Oct 31 12:59:35.781335 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.781149 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 12:59:35.781335 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.781240 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 12:59:35.781335 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.781305 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 12:59:35.839716 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.839686 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcr7\" (UniqueName: \"kubernetes.io/projected/791df070-f034-44c1-8098-df4065c48a32-kube-api-access-4dcr7\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:35.839875 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.839751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/791df070-f034-44c1-8098-df4065c48a32-signing-key\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:35.839875 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.839829 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:35.839875 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.839861 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/791df070-f034-44c1-8098-df4065c48a32-signing-cabundle\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:35.840032 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:35.839978 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:59:35.840089 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:35.840035 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls podName:f338bc5c-18e1-4e62-be2d-93de38d4ae9d nodeName:}" failed. No retries permitted until 2026-10-31 12:59:39.840017093 +0000 UTC m=+36.570630932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls") pod "dns-default-ltf7k" (UID: "f338bc5c-18e1-4e62-be2d-93de38d4ae9d") : secret "dns-default-metrics-tls" not found Oct 31 12:59:35.941137 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.941083 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcr7\" (UniqueName: \"kubernetes.io/projected/791df070-f034-44c1-8098-df4065c48a32-kube-api-access-4dcr7\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:35.941304 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.941189 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/791df070-f034-44c1-8098-df4065c48a32-signing-key\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:35.941304 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.941236 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/791df070-f034-44c1-8098-df4065c48a32-signing-cabundle\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:35.942077 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.942042 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/791df070-f034-44c1-8098-df4065c48a32-signing-cabundle\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:35.944335 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.944304 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/791df070-f034-44c1-8098-df4065c48a32-signing-key\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:35.955814 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:35.955785 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcr7\" (UniqueName: \"kubernetes.io/projected/791df070-f034-44c1-8098-df4065c48a32-kube-api-access-4dcr7\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:36.112826 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:36.112792 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:36.432651 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:36.432595 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6c5t9"] Oct 31 12:59:36.435914 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:36.435890 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791df070_f034_44c1_8098_df4065c48a32.slice/crio-91eb40ca7bc8d9aa20b6abc00d59740e0b982b90506b2131214eeb15da854743 WatchSource:0}: Error finding container 91eb40ca7bc8d9aa20b6abc00d59740e0b982b90506b2131214eeb15da854743: Status 404 returned error can't find the container with id 91eb40ca7bc8d9aa20b6abc00d59740e0b982b90506b2131214eeb15da854743 Oct 31 12:59:36.446392 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:36.446362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:36.446504 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:36.446417 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:36.446543 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:36.446527 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 12:59:36.446604 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:36.446591 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs podName:288087a8-c7ea-40a8-88bb-1b85be0aef55 nodeName:}" failed. No retries permitted until 2026-10-31 13:00:08.446571353 +0000 UTC m=+65.177185198 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs") pod "network-metrics-daemon-sq2q6" (UID: "288087a8-c7ea-40a8-88bb-1b85be0aef55") : secret "metrics-daemon-secret" not found Oct 31 12:59:36.448801 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:36.448775 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:36.547752 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:36.547726 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:36.550032 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:36.550013 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:36.576933 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:36.576914 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:36.801569 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:36.801545 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:37.044811 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:37.044733 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" event={"ID":"791df070-f034-44c1-8098-df4065c48a32","Type":"ContainerStarted","Data":"91eb40ca7bc8d9aa20b6abc00d59740e0b982b90506b2131214eeb15da854743"} Oct 31 12:59:37.046985 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:37.046526 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rm7b9" event={"ID":"9e16e028-6e02-48e1-b54e-3517232c2d8b","Type":"ContainerStarted","Data":"fe738de140a7c5633ddda6a65ec742025117ac8e7b749e96b237c4e14f6970c0"} Oct 31 12:59:37.048900 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:37.048860 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr" event={"ID":"116df32b-c0fe-456f-bea6-2fdf33dc0039","Type":"ContainerStarted","Data":"1d14eb54556f4f1b7c422bd5f0537e6f94a5b771112265eba47bb2eb8ed69fd8"} Oct 31 12:59:37.054777 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:37.054721 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w58sk"] Oct 31 12:59:37.057935 ip-10-0-137-206 kubenswrapper[2583]: W1031 12:59:37.057906 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83f70c7_ab6c_43b0_b86d_3d9d935973ec.slice/crio-9625676fab8b03f3f47ce2d0131502d160da8a3200fcae8afc451ce3c96f9647 WatchSource:0}: Error finding container 9625676fab8b03f3f47ce2d0131502d160da8a3200fcae8afc451ce3c96f9647: Status 404 returned error can't find the container with id 9625676fab8b03f3f47ce2d0131502d160da8a3200fcae8afc451ce3c96f9647 Oct 31 12:59:37.091783 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:37.091733 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr" podStartSLOduration=1.8327323679999998 podStartE2EDuration="4.091716784s" totalImagesPullingTime="2.258984416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:37.090977185 +0000 UTC m=+33.821591048" watchObservedRunningTime="2026-10-31 12:59:37.091716784 +0000 UTC m=+33.822330647" Oct 31 12:59:37.092531 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:37.092387 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rm7b9" podStartSLOduration=1.038814264 podStartE2EDuration="5.092380143s" totalImagesPullingTime="4.053565879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:37.073330258 +0000 UTC m=+33.803944120" watchObservedRunningTime="2026-10-31 12:59:37.092380143 +0000 UTC m=+33.822994005" Oct 31 12:59:37.107702 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:37.107676 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sjsfh"] Oct 31 12:59:37.153045 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:37.153017 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:37.153461 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:37.153085 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:37.153802 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:37.153781 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:59:37.153903 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:37.153846 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls podName:7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:41.153826239 +0000 UTC m=+37.884440083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2knk" (UID: "7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400") : secret "insights-runtime-extractor-tls" not found Oct 31 12:59:37.154755 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:37.154738 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 12:59:37.154860 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:37.154790 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm podName:7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:41.154775374 +0000 UTC m=+37.885389231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2knk" (UID: "7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400") : configmap "kube-rbac-proxy" not found Oct 31 12:59:38.054005 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:38.053968 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8jf6-vd797" event={"ID":"c6266aa7-b443-49e9-88a0-987228617831","Type":"ContainerStarted","Data":"d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384"} Oct 31 12:59:38.055833 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:38.055802 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sjsfh" event={"ID":"09700847-fa74-4444-9d56-5a4e45300c79","Type":"ContainerStarted","Data":"342b035e1591ec68c62457ed3e15171758ac269edc598a5cc90174310d6684a5"} Oct 31 12:59:38.057806 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:38.057787 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w58sk" event={"ID":"d83f70c7-ab6c-43b0-b86d-3d9d935973ec","Type":"ContainerStarted","Data":"9625676fab8b03f3f47ce2d0131502d160da8a3200fcae8afc451ce3c96f9647"} Oct 31 12:59:39.874137 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:39.874073 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:39.874580 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:39.874243 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:59:39.874580 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:39.874317 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls podName:f338bc5c-18e1-4e62-be2d-93de38d4ae9d nodeName:}" failed. No retries permitted until 2026-10-31 12:59:47.874296526 +0000 UTC m=+44.604910381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls") pod "dns-default-ltf7k" (UID: "f338bc5c-18e1-4e62-be2d-93de38d4ae9d") : secret "dns-default-metrics-tls" not found Oct 31 12:59:41.068848 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:41.068789 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" event={"ID":"791df070-f034-44c1-8098-df4065c48a32","Type":"ContainerStarted","Data":"acb1d14e8dc3cc6f910481f015bae27c50abff6b0c63aec10ad7ea6bc4260803"} Oct 31 12:59:41.088985 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:41.088936 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-q8jf6-vd797" podStartSLOduration=4.320403636 podStartE2EDuration="8.088917404s" totalImagesPullingTime="3.768513768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:38.074792089 +0000 UTC m=+34.805405952" watchObservedRunningTime="2026-10-31 12:59:41.088917404 +0000 UTC m=+37.819531267" Oct 31 12:59:41.089918 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:41.089880 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" podStartSLOduration=2.11306812 podStartE2EDuration="6.089868484s" totalImagesPullingTime="3.976800364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:41.087752029 +0000 UTC m=+37.818365908" watchObservedRunningTime="2026-10-31 12:59:41.089868484 +0000 UTC m=+37.820482349" Oct 31 12:59:41.183227 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:41.183191 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:41.183362 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:41.183274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:41.183427 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:41.183359 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:59:41.183495 ip-10-0-137-206 kubenswrapper[2583]: E1031 12:59:41.183429 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls podName:7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:49.183409646 +0000 UTC m=+45.914023500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2knk" (UID: "7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400") : secret "insights-runtime-extractor-tls" not found Oct 31 12:59:41.184264 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:41.184238 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:42.072793 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:42.072750 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sjsfh" event={"ID":"09700847-fa74-4444-9d56-5a4e45300c79","Type":"ContainerStarted","Data":"f08a6f470eeed659b9ccb5a0416f12564582bf94dd2402225eb511b5bf0c1b30"} Oct 31 12:59:42.073253 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:42.072998 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:42.093041 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:42.092997 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sjsfh" podStartSLOduration=34.138565042 podStartE2EDuration="38.092982963s" totalImagesPullingTime="3.954417921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:42.092181951 +0000 UTC m=+38.822795813" watchObservedRunningTime="2026-10-31 12:59:42.092982963 +0000 UTC m=+38.823596826" Oct 31 12:59:43.077580 ip-10-0-137-206 kubenswrapper[2583]: I1031 12:59:43.077545 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w58sk" event={"ID":"d83f70c7-ab6c-43b0-b86d-3d9d935973ec","Type":"ContainerStarted","Data":"ef3e7f73b530ab96dcfef988bbcd45dccc752e47edae0e7597a052aa3d1908b1"} Oct 31 12:59:43.085470 ip-10-0-137-206 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 12:59:43.095880 ip-10-0-137-206 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 12:59:43.096057 ip-10-0-137-206 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 12:59:43.096124 ip-10-0-137-206 systemd[1]: kubelet.service: Consumed 2.314s CPU time. Oct 31 12:59:43.111071 ip-10-0-137-206 systemd[1]: Starting Kubernetes Kubelet... Oct 31 12:59:43.203305 ip-10-0-137-206 kubenswrapper[4542]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:59:43.203305 ip-10-0-137-206 kubenswrapper[4542]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 12:59:43.203305 ip-10-0-137-206 kubenswrapper[4542]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:59:43.203305 ip-10-0-137-206 kubenswrapper[4542]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:59:43.205364 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205295 4542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:43.205364 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205363 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205368 4542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205371 4542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205375 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205377 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205380 4542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205383 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205385 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205388 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205391 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205393 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205396 4542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205398 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205401 4542 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205404 4542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205406 4542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205409 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205412 4542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205414 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:43.205430 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205417 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205425 4542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205427 4542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205431 4542 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205434 4542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205437 4542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205439 4542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205442 4542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205445 4542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205448 4542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205450 4542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205453 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205456 4542 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205458 4542 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205461 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205463 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205466 4542 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205468 4542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205471 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205486 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:43.205895 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205489 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205492 4542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205496 4542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205500 4542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205503 4542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205506 4542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205510 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205513 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205516 4542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205519 4542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205522 4542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205524 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205527 4542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205530 4542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205533 4542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205535 4542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205538 4542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205540 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205543 4542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:43.206376 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205545 4542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205550 4542 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205553 4542 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205556 4542 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205559 4542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205563 4542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205565 4542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205568 4542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205571 4542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205574 4542 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205576 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205580 4542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205584 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205588 4542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205591 4542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205594 4542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205597 4542 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205599 4542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205602 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205608 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:43.206846 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205611 4542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205613 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205616 4542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205619 4542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205622 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205624 4542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205627 4542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205630 4542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205632 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205635 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205638 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205641 4542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205643 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205646 4542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205648 4542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205651 4542 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205654 4542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205657 4542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205660 4542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205663 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:43.207328 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205665 4542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205668 4542 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205670 4542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205673 4542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205676 4542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205678 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.205681 4542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206432 4542 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206440 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206443 4542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206446 4542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206451 4542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206454 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206457 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206460 4542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206463 4542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206466 4542 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206469 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206471 4542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206488 4542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:43.207813 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206491 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206494 4542 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206497 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206499 4542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206502 4542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206505 4542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206508 4542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206510 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206513 4542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206516 4542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206519 4542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206521 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206524 4542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206526 4542 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206529 4542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206532 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206534 4542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206537 4542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206539 4542 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:43.208290 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206543 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206546 4542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206549 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206552 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206556 4542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206559 4542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206561 4542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206564 4542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206567 4542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206569 4542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206572 4542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206574 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206577 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206580 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206582 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206585 4542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206587 4542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206590 4542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206593 4542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:43.208768 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206596 4542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206598 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206601 4542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206604 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206607 4542 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206610 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206612 4542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206615 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206619 4542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206623 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206626 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206629 4542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206632 4542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206635 4542 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206638 4542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206641 4542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206644 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206648 4542 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206651 4542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206653 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:43.209232 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206656 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206658 4542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206661 4542 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206664 4542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206666 4542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206669 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206671 4542 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206674 4542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206677 4542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206679 4542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206682 4542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206684 4542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206687 4542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206689 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206692 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206695 4542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206697 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206701 4542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206705 4542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:43.209725 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206708 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206711 4542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206714 4542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206717 4542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206720 4542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206723 4542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206727 4542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206730 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206733 4542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206736 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206739 4542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206742 4542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206745 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206748 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206750 4542 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.206753 4542 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207243 4542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207249 4542 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207252 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207255 4542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:43.210216 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207258 4542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207261 4542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207264 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207267 4542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207270 4542 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207272 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207275 4542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207278 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207281 4542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207283 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207286 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207288 4542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207291 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207294 4542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207296 4542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207299 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207301 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207304 4542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207307 4542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:43.210703 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207310 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207313 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207315 4542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207320 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207322 4542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207325 4542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207328 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207330 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207333 4542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207336 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207339 4542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207341 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207344 4542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207346 4542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207350 4542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207352 4542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207355 4542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207357 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207360 4542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207362 4542 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:43.211278 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207366 4542 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207369 4542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207371 4542 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207374 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207376 4542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207379 4542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207381 4542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207384 4542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207386 4542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207389 4542 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207391 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207394 4542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207397 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207399 4542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207404 4542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207409 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207412 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207415 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207418 4542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207421 4542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:43.212143 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207423 4542 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207426 4542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207429 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207431 4542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207434 4542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207437 4542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207439 4542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207442 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207444 4542 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207447 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207450 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207453 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207456 4542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207458 4542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207461 4542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207463 4542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207466 4542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207469 4542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207471 4542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207486 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:43.213005 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207489 4542 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207492 4542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207494 4542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207497 4542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207499 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207502 4542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207505 4542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207524 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207528 4542 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207530 4542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207533 4542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207536 4542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207538 4542 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207541 4542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207544 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207548 4542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207552 4542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207555 4542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207558 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:43.214018 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207561 4542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207563 4542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207566 4542 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.207569 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207665 4542 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207681 4542 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207690 4542 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207697 4542 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207702 4542 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207706 4542 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207710 4542 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207714 4542 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207718 4542 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207721 4542 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207724 4542 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207727 4542 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207731 4542 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207733 4542 flags.go:64] FLAG: --cgroup-root="" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207737 4542 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207740 4542 flags.go:64] FLAG: --client-ca-file="" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207745 4542 flags.go:64] FLAG: --cloud-provider="external" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207749 4542 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207753 4542 flags.go:64] FLAG: --cluster-domain="" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207756 4542 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 12:59:43.214609 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207759 4542 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207762 4542 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207766 4542 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207770 4542 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207773 4542 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207776 4542 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207779 4542 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207782 4542 flags.go:64] FLAG: --contention-profiling="false" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207785 4542 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207788 4542 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207792 4542 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207795 4542 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207799 4542 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207802 4542 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207805 4542 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207808 4542 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207812 4542 flags.go:64] FLAG: --enable-server="true" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207815 4542 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207819 4542 flags.go:64] FLAG: --event-burst="100" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207823 4542 flags.go:64] FLAG: --event-qps="50" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207826 4542 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207829 4542 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207832 4542 flags.go:64] FLAG: --eviction-hard="" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207836 4542 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 12:59:43.215173 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207839 4542 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207842 4542 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207845 4542 flags.go:64] FLAG: --eviction-soft="" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207848 4542 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207852 4542 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207855 4542 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207858 4542 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207862 4542 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207865 4542 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207867 4542 flags.go:64] FLAG: --feature-gates="" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207873 4542 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207876 4542 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207880 4542 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207883 4542 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207887 4542 flags.go:64] FLAG: --healthz-port="10248" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207890 4542 flags.go:64] FLAG: --help="false" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207893 4542 flags.go:64] FLAG: --hostname-override="ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207897 4542 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207900 4542 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207903 4542 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207906 4542 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207909 4542 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207912 4542 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 12:59:43.215805 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207915 4542 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207918 4542 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207921 4542 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207924 4542 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207927 4542 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207930 4542 flags.go:64] FLAG: --kube-reserved="" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207933 4542 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207935 4542 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207938 4542 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207941 4542 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207944 4542 flags.go:64] FLAG: --lock-file="" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207947 4542 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207950 4542 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207953 4542 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207959 4542 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207962 4542 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207966 4542 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207969 4542 flags.go:64] FLAG: --logging-format="text" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207972 4542 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207977 4542 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207980 4542 flags.go:64] FLAG: --manifest-url="" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207983 4542 flags.go:64] FLAG: --manifest-url-header="" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207988 4542 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207991 4542 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207995 4542 flags.go:64] FLAG: --max-pods="110" Oct 31 12:59:43.216397 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.207998 4542 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208002 4542 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208005 4542 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208008 4542 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208011 4542 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208014 4542 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208017 4542 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208023 4542 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208027 4542 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208030 4542 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208033 4542 flags.go:64] FLAG: --pod-cidr="" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208036 4542 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208039 4542 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208042 4542 flags.go:64] FLAG: --pods-per-core="0" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208045 4542 flags.go:64] FLAG: --port="10250" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208048 4542 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208051 4542 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08340c8f5e8ff6077" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208054 4542 flags.go:64] FLAG: --qos-reserved="" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208057 4542 flags.go:64] FLAG: --read-only-port="10255" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208060 4542 flags.go:64] FLAG: --register-node="true" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208063 4542 flags.go:64] FLAG: --register-with-taints="" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208066 4542 flags.go:64] FLAG: --registry-burst="10" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208069 4542 flags.go:64] FLAG: --registry-qps="5" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208072 4542 flags.go:64] FLAG: --reserved-cpus="" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208075 4542 flags.go:64] FLAG: --reserved-memory="" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208079 4542 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 12:59:43.217024 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208083 4542 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208086 4542 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208089 4542 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208092 4542 flags.go:64] FLAG: --runonce="false" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208095 4542 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208098 4542 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208102 4542 flags.go:64] FLAG: --seccomp-default="false" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208105 4542 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208107 4542 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208111 4542 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208114 4542 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208117 4542 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208120 4542 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208123 4542 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208126 4542 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208129 4542 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208132 4542 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208135 4542 flags.go:64] FLAG: --system-cgroups="" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208138 4542 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208143 4542 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208146 4542 flags.go:64] FLAG: --tls-cert-file="" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208149 4542 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208153 4542 flags.go:64] FLAG: --tls-min-version="" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208156 4542 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208158 4542 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 12:59:43.217656 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208161 4542 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208164 4542 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208167 4542 flags.go:64] FLAG: --v="2" Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208172 4542 flags.go:64] FLAG: --version="false" Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208176 4542 flags.go:64] FLAG: --vmodule="" Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208180 4542 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208183 4542 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208355 4542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208359 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208362 4542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208365 4542 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208369 4542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208371 4542 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208374 4542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208377 4542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208380 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208382 4542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208385 4542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208388 4542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208391 4542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208393 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:43.218235 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208396 4542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208399 4542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208401 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208404 4542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208407 4542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208409 4542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208412 4542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208415 4542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208417 4542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208420 4542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208422 4542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208425 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208428 4542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208431 4542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208435 4542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208439 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208442 4542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208445 4542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208450 4542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:43.218744 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208452 4542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208455 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208457 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208461 4542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208464 4542 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208467 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208470 4542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208485 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208488 4542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208491 4542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208494 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208496 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208499 4542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208502 4542 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208504 4542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208507 4542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208510 4542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208513 4542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208515 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208518 4542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:43.219260 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208521 4542 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208523 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208526 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208528 4542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208531 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208534 4542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208536 4542 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208539 4542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208541 4542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208544 4542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208547 4542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208551 4542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208554 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208556 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208559 4542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208563 4542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208566 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208568 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208572 4542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:43.219884 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208575 4542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208578 4542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208582 4542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208585 4542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208588 4542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208602 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208605 4542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208608 4542 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208611 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208614 4542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208616 4542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208619 4542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208622 4542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208624 4542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208627 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208630 4542 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208633 4542 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208636 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208639 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208642 4542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:43.220509 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208644 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208647 4542 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208649 4542 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208652 4542 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208656 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208659 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208662 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208665 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208669 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208672 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208675 4542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208677 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208680 4542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208682 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.208711 4542 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208754 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:43.221003 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208758 4542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208762 4542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208764 4542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208767 4542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208770 4542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208773 4542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208775 4542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208779 4542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208782 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208784 4542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208787 4542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208791 4542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208794 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208797 4542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208800 4542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208802 4542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208805 4542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208807 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:43.221402 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208810 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208812 4542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208816 4542 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208819 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208822 4542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208825 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208827 4542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208831 4542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208834 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208837 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208840 4542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208842 4542 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208845 4542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208847 4542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208850 4542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208853 4542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208855 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208858 4542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208861 4542 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208863 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:43.222138 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208866 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208868 4542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208871 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208874 4542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208876 4542 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208879 4542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208881 4542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208884 4542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208886 4542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208889 4542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208891 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208894 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208897 4542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208899 4542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208903 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208905 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208910 4542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208914 4542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208917 4542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:43.222752 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208921 4542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208924 4542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208926 4542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208929 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208931 4542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208934 4542 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208937 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208939 4542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208942 4542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208944 4542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208947 4542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208950 4542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208952 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208955 4542 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208957 4542 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208960 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208963 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208965 4542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208968 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208971 4542 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:43.223202 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208974 4542 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208976 4542 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208979 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208981 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208984 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208986 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208989 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208993 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208996 4542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.208999 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209001 4542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209004 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209008 4542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209011 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209014 4542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209016 4542 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209019 4542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209021 4542 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209024 4542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209027 4542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:43.223694 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209029 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209032 4542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209034 4542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209037 4542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209040 4542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209042 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209045 4542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.209047 4542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.213942 4542 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.213967 4542 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.213991 4542 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.213999 4542 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214662 4542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214777 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214782 4542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214785 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214788 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214790 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214794 4542 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:43.224170 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214797 4542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214799 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214805 4542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214808 4542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214812 4542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214815 4542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214818 4542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214821 4542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214824 4542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214827 4542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214830 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214833 4542 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214836 4542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214838 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214841 4542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214844 4542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214847 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214849 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214852 4542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:43.224655 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214854 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214857 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214860 4542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214862 4542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214865 4542 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214868 4542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214871 4542 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214873 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214876 4542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214879 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214881 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214884 4542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214887 4542 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214889 4542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214892 4542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214894 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214897 4542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214899 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214902 4542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214905 4542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:43.225131 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214908 4542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214910 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214914 4542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214918 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214921 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214924 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214926 4542 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214929 4542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214931 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214934 4542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214936 4542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214939 4542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214942 4542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214945 4542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214947 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214950 4542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214953 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214955 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214958 4542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:43.225764 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214961 4542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214964 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214966 4542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214969 4542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214971 4542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214974 4542 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214977 4542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214980 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214982 4542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214985 4542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214988 4542 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214991 4542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214994 4542 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214996 4542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.214999 4542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215001 4542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215004 4542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215007 4542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215009 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:43.226206 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215012 4542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215014 4542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215017 4542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215020 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215022 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215025 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215028 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215030 4542 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215033 4542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215035 4542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215038 4542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215041 4542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215043 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215046 4542 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215049 4542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215051 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215054 4542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215056 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215059 4542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215061 4542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:43.226690 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215064 4542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215067 4542 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.215103 4542 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215247 4542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215252 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215255 4542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215258 4542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215261 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215265 4542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215268 4542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215271 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215276 4542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215279 4542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215283 4542 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215285 4542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215288 4542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:43.227190 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215291 4542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215293 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215296 4542 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215299 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215301 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215304 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215307 4542 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215310 4542 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215312 4542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215315 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215318 4542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215321 4542 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215323 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215326 4542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215328 4542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215331 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215333 4542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215336 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215339 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215341 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:43.227593 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215344 4542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215346 4542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215349 4542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215352 4542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215355 4542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215360 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215362 4542 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215365 4542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215368 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215370 4542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215373 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215375 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215378 4542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215381 4542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215383 4542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215386 4542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215389 4542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215391 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215394 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:43.228064 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215397 4542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215399 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215402 4542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215404 4542 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215407 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215410 4542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215412 4542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215415 4542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215418 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215420 4542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215423 4542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215426 4542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215428 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215431 4542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215435 4542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215439 4542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215442 4542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215444 4542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215451 4542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:43.228638 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215454 4542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215457 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215459 4542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215462 4542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215464 4542 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215467 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215469 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215488 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215492 4542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215494 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215497 4542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215500 4542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215503 4542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215506 4542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215508 4542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215511 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215513 4542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215516 4542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215519 4542 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215522 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:43.229095 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215524 4542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215527 4542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215530 4542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215532 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215535 4542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215538 4542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215541 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215543 4542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215546 4542 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215548 4542 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215551 4542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215555 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215557 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215561 4542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215564 4542 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.215589 4542 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:59:43.229837 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215641 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215645 4542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215648 4542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215651 4542 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215654 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215657 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215661 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215664 4542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215667 4542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215669 4542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215672 4542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215675 4542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215678 4542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215680 4542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215683 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215686 4542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215689 4542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215691 4542 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215694 4542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215696 4542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:59:43.230469 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215699 4542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215702 4542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215704 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215707 4542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215709 4542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215712 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215715 4542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215717 4542 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215721 4542 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215724 4542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215730 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215733 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215736 4542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215738 4542 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215741 4542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215744 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215746 4542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215749 4542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215753 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:59:43.230969 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215755 4542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215758 4542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215760 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215765 4542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215768 4542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215771 4542 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215774 4542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215777 4542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215779 4542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215782 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215785 4542 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215787 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215790 4542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215792 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215795 4542 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215798 4542 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215800 4542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215804 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215807 4542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215810 4542 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:59:43.231419 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215812 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215816 4542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215818 4542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215821 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215829 4542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215833 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215835 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215838 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215840 4542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215843 4542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215845 4542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215849 4542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215852 4542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215855 4542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215857 4542 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215860 4542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215863 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215865 4542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215868 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:59:43.231934 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215870 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215873 4542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215876 4542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215878 4542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215881 4542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215883 4542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215885 4542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215888 4542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215891 4542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215893 4542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215896 4542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215898 4542 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215901 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215903 4542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215906 4542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215909 4542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215911 4542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215914 4542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215917 4542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215920 4542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:59:43.232602 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215923 4542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215925 4542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215930 4542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215934 4542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215938 4542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215941 4542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215944 4542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:43.215947 4542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.215998 4542 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.216951 4542 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.217004 4542 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.217184 4542 server.go:1014] "Starting client certificate rotation" Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.217323 4542 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.217381 4542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:54:03 +0000 UTC" deadline="2028-06-13 20:40:24.793195723 +0000 UTC" Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.217415 4542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14191h40m41.57578285s" Oct 31 12:59:43.233262 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.217657 4542 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:59:43.233665 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.218731 4542 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:59:43.233665 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.219880 4542 log.go:25] "Validated CRI v1 runtime API" Oct 31 12:59:43.233665 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.221559 4542 log.go:25] "Validated CRI v1 image API" Oct 31 12:59:43.233665 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.222287 4542 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 12:59:43.233665 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.223778 4542 fs.go:135] Filesystem UUIDs: map[300f6a1f-be2a-4cdb-9abf-9a5901ab94c2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 aabf960d-ae26-4290-aea0-4b954e44ae1a:/dev/nvme0n1p4] Oct 31 12:59:43.233828 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.223792 4542 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06eaee4b4a0d86a988b1a795d07f34a7698d5428b6c43e4622134721248570fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06eaee4b4a0d86a988b1a795d07f34a7698d5428b6c43e4622134721248570fc/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20e9d7a3ff5e237a1dbc6bd4d5b64abf01a99db44b7a25bf2e0835d373536346/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20e9d7a3ff5e237a1dbc6bd4d5b64abf01a99db44b7a25bf2e0835d373536346/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/256e47eee838717b2a5550e39abbf29ab90699afeb6eb49bd5be65e316afe491/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/256e47eee838717b2a5550e39abbf29ab90699afeb6eb49bd5be65e316afe491/userdata/shm major:0 minor:54 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/342b035e1591ec68c62457ed3e15171758ac269edc598a5cc90174310d6684a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/342b035e1591ec68c62457ed3e15171758ac269edc598a5cc90174310d6684a5/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4614d6bfe74b19056ef5e1c8e436af77da12792b1c2abdad477e3cf7025a7d60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4614d6bfe74b19056ef5e1c8e436af77da12792b1c2abdad477e3cf7025a7d60/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f9b48e55549ee1f76bb3a9f20a3e727fab4a390369a40f2599574357d79baca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f9b48e55549ee1f76bb3a9f20a3e727fab4a390369a40f2599574357d79baca/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/616e70f1a66a6f876a1ba23e7a895523dc3c488c3dc17c2a36e0d2d5e260fbe5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/616e70f1a66a6f876a1ba23e7a895523dc3c488c3dc17c2a36e0d2d5e260fbe5/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ba7ad2e2d6a4d8240b1441318185d91836bd5fc221bf0d71130a7dc20ef5b7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ba7ad2e2d6a4d8240b1441318185d91836bd5fc221bf0d71130a7dc20ef5b7a/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c57f50c7d7ef264584224c6e6077cfaea3f1c5aaa7c7518b5e4f0fe81d94de8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c57f50c7d7ef264584224c6e6077cfaea3f1c5aaa7c7518b5e4f0fe81d94de8/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d0156f0bb202ee1e9a13702d1b9662569f6daa6596a371330024765aa21c1d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d0156f0bb202ee1e9a13702d1b9662569f6daa6596a371330024765aa21c1d9/userdata/shm major:0 minor:53 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91eb40ca7bc8d9aa20b6abc00d59740e0b982b90506b2131214eeb15da854743/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91eb40ca7bc8d9aa20b6abc00d59740e0b982b90506b2131214eeb15da854743/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9625676fab8b03f3f47ce2d0131502d160da8a3200fcae8afc451ce3c96f9647/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9625676fab8b03f3f47ce2d0131502d160da8a3200fcae8afc451ce3c96f9647/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb6bfebd88384d0f5871c17765d681316d4abaed0ed6190bcea80caa997a96ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb6bfebd88384d0f5871c17765d681316d4abaed0ed6190bcea80caa997a96ef/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4fb16a264a1a0b4ad825bf196dceaecf7ec96056a69bca47f4a6d8658bad448/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4fb16a264a1a0b4ad825bf196dceaecf7ec96056a69bca47f4a6d8658bad448/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d79bd8837dd89a25bb7e97a9c72976dc77173457e3393580df8cfadd3e540867/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d79bd8837dd89a25bb7e97a9c72976dc77173457e3393580df8cfadd3e540867/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9fcedf6d05e9ccfdd12ff1235c323d6faa9d63db4729007e0adce5463026ad3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9fcedf6d05e9ccfdd12ff1235c323d6faa9d63db4729007e0adce5463026ad3/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09700847-fa74-4444-9d56-5a4e45300c79/volumes/kubernetes.io~projected/kube-api-access-r8j2z:{mountpoint:/var/lib/kubelet/pods/09700847-fa74-4444-9d56-5a4e45300c79/volumes/kubernetes.io~projected/kube-api-access-r8j2z major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/116df32b-c0fe-456f-bea6-2fdf33dc0039/volumes/kubernetes.io~projected/kube-api-access-plv7c:{mountpoint:/var/lib/kubelet/pods/116df32b-c0fe-456f-bea6-2fdf33dc0039/volumes/kubernetes.io~projected/kube-api-access-plv7c major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1aaae2c3-0ea2-4061-a448-e50f82dcf823/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1aaae2c3-0ea2-4061-a448-e50f82dcf823/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/288087a8-c7ea-40a8-88bb-1b85be0aef55/volumes/kubernetes.io~projected/kube-api-access-tgf48:{mountpoint:/var/lib/kubelet/pods/288087a8-c7ea-40a8-88bb-1b85be0aef55/volumes/kubernetes.io~projected/kube-api-access-tgf48 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d525238-6ced-4f9f-89a6-56fb50fba909/volumes/kubernetes.io~projected/kube-api-access-jtwpx:{mountpoint:/var/lib/kubelet/pods/5d525238-6ced-4f9f-89a6-56fb50fba909/volumes/kubernetes.io~projected/kube-api-access-jtwpx major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/791df070-f034-44c1-8098-df4065c48a32/volumes/kubernetes.io~projected/kube-api-access-4dcr7:{mountpoint:/var/lib/kubelet/pods/791df070-f034-44c1-8098-df4065c48a32/volumes/kubernetes.io~projected/kube-api-access-4dcr7 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/791df070-f034-44c1-8098-df4065c48a32/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/791df070-f034-44c1-8098-df4065c48a32/volumes/kubernetes.io~secret/signing-key major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bf98082-9c6b-44e8-99e1-8ad8320b3483/volumes/kubernetes.io~projected/kube-api-access-bw5pd:{mountpoint:/var/lib/kubelet/pods/7bf98082-9c6b-44e8-99e1-8ad8320b3483/volumes/kubernetes.io~projected/kube-api-access-bw5pd major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400/volumes/kubernetes.io~projected/kube-api-access-xjnj2:{mountpoint:/var/lib/kubelet/pods/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400/volumes/kubernetes.io~projected/kube-api-access-xjnj2 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e16e028-6e02-48e1-b54e-3517232c2d8b/volumes/kubernetes.io~projected/kube-api-access-872ln:{mountpoint:/var/lib/kubelet/pods/9e16e028-6e02-48e1-b54e-3517232c2d8b/volumes/kubernetes.io~projected/kube-api-access-872ln major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e16e028-6e02-48e1-b54e-3517232c2d8b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9e16e028-6e02-48e1-b54e-3517232c2d8b/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5a2f7b9-0ca0-4670-80cd-16e31aecb597/volumes/kubernetes.io~projected/kube-api-access-mll9t:{mountpoint:/var/lib/kubelet/pods/a5a2f7b9-0ca0-4670-80cd-16e31aecb597/volumes/kubernetes.io~projected/kube-api-access-mll9t major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6266aa7-b443-49e9-88a0-987228617831/volumes/kubernetes.io~projected/kube-api-access-z4fm9:{mountpoint:/var/lib/kubelet/pods/c6266aa7-b443-49e9-88a0-987228617831/volumes/kubernetes.io~projected/kube-api-access-z4fm9 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cabced46-3402-4172-aa1b-f056b9e01220/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/cabced46-3402-4172-aa1b-f056b9e01220/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cabced46-3402-4172-aa1b-f056b9e01220/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/cabced46-3402-4172-aa1b-f056b9e01220/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cabced46-3402-4172-aa1b-f056b9e01220/volumes/kubernetes.io~projected/kube-api-access-xfpfl:{mountpoint:/var/lib/kubelet/pods/cabced46-3402-4172-aa1b-f056b9e01220/volumes/kubernetes.io~projected/kube-api-access-xfpfl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d83f70c7-ab6c-43b0-b86d-3d9d935973ec/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d83f70c7-ab6c-43b0-b86d-3d9d935973ec/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f/volumes/kubernetes.io~projected/kube-api-access-ncmqn:{mountpoint:/var/lib/kubelet/pods/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f/volumes/kubernetes.io~projected/kube-api-access-ncmqn major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed98cbbd-063e-43ed-bddc-ba216fbf56bb/volumes/kubernetes.io~projected/kube-api-access-w9hhh:{mountpoint:/var/lib/kubelet/pods/ed98cbbd-063e-43ed-bddc-ba216fbf56bb/volumes/kubernetes.io~projected/kube-api-access-w9hhh major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef470cac-0cd7-4fae-87e2-f6d431227d00/volumes/kubernetes.io~projected/kube-api-access-d7wgc:{mountpoint:/var/lib/kubelet/pods/ef470cac-0cd7-4fae-87e2-f6d431227d00/volumes/kubernetes.io~projected/kube-api-access-d7wgc major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f338bc5c-18e1-4e62-be2d-93de38d4ae9d/volumes/kubernetes.io~projected/kube-api-access-x65pj:{mountpoint:/var/lib/kubelet/pods/f338bc5c-18e1-4e62-be2d-93de38d4ae9d/volumes/kubernetes.io~projected/kube-api-access-x65pj major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa9e011c-a51f-4738-b970-58ec77045b81/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fa9e011c-a51f-4738-b970-58ec77045b81/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa9e011c-a51f-4738-b970-58ec77045b81/volumes/kubernetes.io~projected/kube-api-access-svj5d:{mountpoint:/var/lib/kubelet/pods/fa9e011c-a51f-4738-b970-58ec77045b81/volumes/kubernetes.io~projected/kube-api-access-svj5d major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa9e011c-a51f-4738-b970-58ec77045b81/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fa9e011c-a51f-4738-b970-58ec77045b81/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/05a7d2b0aa69f71b4ee4ed95d7a3d134057709d5396e1bcabc06fe8b1caa9a5d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9301816e2135cfc5147e1774c6cf74151c818dd2d52e953562f4b385597f19da/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8820296dd1b917544ee3fef1c0e27ce69b7c87e85da68fc5ad66b0d685fed7ce/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ceb146e1923f271da7e803af358856f77129487385249f0ccc192bf575e23531/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b2e499d7973ef7ced45ef7dc8300d7ac5fcc1b87d37acec4f698833328fa67a7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/171a4fde0584650e686ef2c39e2da4f10a5e3df8d32888632f86921cf11dc484/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4fac83980bed43e9d982760ef9104cd5cb2befadb341f8aa60ba4eb0f722be4d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0416e7b58902933476275611e926e27dc0ced1187294bde4c2408f3e2fdb1cd8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1fbd280a2c74b3219183d7be295d1fcaa36c2365970985ba1ddcf24e223af47c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f6da06a7e998067d25f7e96a932cc6abc556cf4ebfa0f04b79e760d148d956f0/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/ec03d39b86b29730cbdecf2d7c56c30a740858dea11aeeb2062e743527f814b4/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/06b53f9813e6968f1d0daa8dd2aaea31a7bbb84442f4f14bd4a755a30ba3768d/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/201dec2093fc07f3cca07b903dbdb4c21168fc8a4c7a71341bc8950a6eb8247d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/d4c4f9a1322b0e8a59d8aeaa643e15fb7b10a0811b006702d875b2ff58d7a859/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/8c59f410d8dbd148ca5c18d6b1dbeb6ca7e4faa58fb9035d0aabda65f2957b56/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/83bd751817d2a2729c608a1323782d3b42f60dff6e0afeed0ea3e9dc2a06ff81/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d90c20c2c32e1fcf9806532e4fcdd9afa05f6aaa49fee500256a7b6d8482850b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a1e3310c595becedee99c95bee681bda6fdc8fcb53ca3542cc5f90da2120d980/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d26df8734a919d34fba4638b268d3d01c3da4f34a07a60cd1ba3c8a265a353f6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f22dd1805c67c60a8443bab5009e7dc52441660db30f55c250c4fe8d45de32b8/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2a53ad8307741d2128186f55158df975b98fd3a0d4f3c70007376110c0dde68f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/ae12c7d78f591ea611865791526dbf110490a1e17bf49d737f4fb83b5420c5d7/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/66546af5ab66a4797ff9ccb8725cfea8dc8e8022b0dcd94222d03702cf82e3f3/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/88edf783a8755f2230b0384cc37c842b168bb43378a93924949071a5755fa692/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/31c182f84182f9f4ef65f0334fbf0af414f35f44c60d6119e4d471b8f61b69f6/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/23a14c78fb135441bda7b68a9bf8a18444e31a622f84f6aed1ba3cbdf41dab87/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/e5431e07c5d3baadfbe9812c80759f9e1cc7ae6e55942b4f8eeb7421219d70c2/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/088654171ceeb387fa6420e8f264cd3ea7ca3962df2dc22969258a636d4f153d/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/5a3c362739c9f4a04836220657c78ed7185d7bfa18743a2ce6e24a9729e9189c/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/54c8b325d9930d8d25f1a62f94f607cf8477c383086452a32ff3776d553373a5/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/d6b54ecb64ae206f801aa6355e0a07f9ffa5808e77afa42d798ad2006a5933e8/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/8da058e35daa7c1ee912e047dd43d909740aad71e6c1933521ab0771feef14d6/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/05daa81161baf10f6c8616d7161784d86fd23e78a8ff971ace59031a3418a570/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-59:{mountpoint:/var/lib/containers/storage/overlay/0867cf21736aafb297c70850afc979c83084cdd2c546c435ee87bd56ef775724/merged major:0 minor:59 fsType:overlay blockSize:0} overlay_0-61:{mountpoint:/var/lib/containers/storage/overlay/1cb332e423cb6fff33701ccbc6248262d18f1af9881bd527724de964ae417579/merged major:0 minor:61 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ba526b4348cbf46ef10a0bc7a968c68e65ef0571f2bf893e6d6acc8d788be357/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/b4e585fdf394fb587e2b22fb61f1cb7c2a0b9598bc0616fb65cd97ec72e28cda/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6ca6d87f4ce19f5f6d60407f12647f27645bb7c8b156b3d30f9c28b2640ab6f1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3c1a9bb9d4ae1b02099b3a278fb110e442abd1dfe82435a12b88b6318062ea90/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/606a1b286d2b35ee51f24f522c97cba098ef49334e36be7b029428cdf85d64ff/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/7d5e0d688b0ef03ad22fae4c0296959fe8f555ebae9a8d3812bbd50574619c6c/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/bee3837543cae44dee0611373f53c95b980db1cb290cfc8928f236baab1620da/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5a047feeb4d62d7d6d9db36ec0aa97bb5de6d64d89d1bbb34412da81c2877380/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.230624 4542 manager.go:217] Machine: {Timestamp:2026-10-31 12:59:43.229661245 +0000 UTC m=+0.074827655 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100078 MemoryCapacity:33164320768 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25468284c75cf359c37b4b40288bf2 SystemUUID:ec254682-84c7-5cf3-59c3-7b4b40288bf2 BootID:d8f72d01-f47d-457b-98c0-0265a10b5dde Filesystems:[{Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef470cac-0cd7-4fae-87e2-f6d431227d00/volumes/kubernetes.io~projected/kube-api-access-d7wgc DeviceMajor:0 DeviceMinor:67 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91eb40ca7bc8d9aa20b6abc00d59740e0b982b90506b2131214eeb15da854743/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa9e011c-a51f-4738-b970-58ec77045b81/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6266aa7-b443-49e9-88a0-987228617831/volumes/kubernetes.io~projected/kube-api-access-z4fm9 DeviceMajor:0 DeviceMinor:206 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/342b035e1591ec68c62457ed3e15171758ac269edc598a5cc90174310d6684a5/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/fa9e011c-a51f-4738-b970-58ec77045b81/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/5d525238-6ced-4f9f-89a6-56fb50fba909/volumes/kubernetes.io~projected/kube-api-access-jtwpx DeviceMajor:0 DeviceMinor:63 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f9b48e55549ee1f76bb3a9f20a3e727fab4a390369a40f2599574357d79baca/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-59 DeviceMajor:0 DeviceMinor:59 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06eaee4b4a0d86a988b1a795d07f34a7698d5428b6c43e4622134721248570fc/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/cabced46-3402-4172-aa1b-f056b9e01220/volumes/kubernetes.io~projected/kube-api-access-xfpfl DeviceMajor:0 DeviceMinor:65 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20e9d7a3ff5e237a1dbc6bd4d5b64abf01a99db44b7a25bf2e0835d373536346/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1aaae2c3-0ea2-4061-a448-e50f82dcf823/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f/volumes/kubernetes.io~projected/kube-api-access-ncmqn DeviceMajor:0 DeviceMinor:52 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/a5a2f7b9-0ca0-4670-80cd-16e31aecb597/volumes/kubernetes.io~projected/kube-api-access-mll9t DeviceMajor:0 DeviceMinor:64 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7bf98082-9c6b-44e8-99e1-8ad8320b3483/volumes/kubernetes.io~projected/kube-api-access-bw5pd DeviceMajor:0 DeviceMinor:66 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cabced46-3402-4172-aa1b-f056b9e01220/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-61 DeviceMajor:0 DeviceMinor:61 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb6bfebd88384d0f5871c17765d681316d4abaed0ed6190bcea80caa997a96ef/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4fb16a264a1a0b4ad825bf196dceaecf7ec96056a69bca47f4a6d8658bad448/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cabced46-3402-4172-aa1b-f056b9e01220/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d83f70c7-ab6c-43b0-b86d-3d9d935973ec/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:223 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e16e028-6e02-48e1-b54e-3517232c2d8b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ba7ad2e2d6a4d8240b1441318185d91836bd5fc221bf0d71130a7dc20ef5b7a/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/791df070-f034-44c1-8098-df4065c48a32/volumes/kubernetes.io~projected/kube-api-access-4dcr7 DeviceMajor:0 DeviceMinor:218 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/fa9e011c-a51f-4738-b970-58ec77045b81/volumes/kubernetes.io~projected/kube-api-access-svj5d DeviceMajor:0 DeviceMinor:50 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c57f50c7d7ef264584224c6e6077cfaea3f1c5aaa7c7518b5e4f0fe81d94de8/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/791df070-f034-44c1-8098-df4065c48a32/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:217 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9625676fab8b03f3f47ce2d0131502d160da8a3200fcae8afc451ce3c96f9647/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4614d6bfe74b19056ef5e1c8e436af77da12792b1c2abdad477e3cf7025a7d60/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e16e028-6e02-48e1-b54e-3517232c2d8b/volumes/kubernetes.io~projected/kube-api-access-872ln DeviceMajor:0 DeviceMinor:197 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582160384 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9fcedf6d05e9ccfdd12ff1235c323d6faa9d63db4729007e0adce5463026ad3/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400/volumes/kubernetes.io~projected/kube-api-access-xjnj2 DeviceMajor:0 DeviceMinor:207 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d79bd8837dd89a25bb7e97a9c72976dc77173457e3393580df8cfadd3e540867/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/09700847-fa74-4444-9d56-5a4e45300c79/volumes/kubernetes.io~projected/kube-api-access-r8j2z DeviceMajor:0 DeviceMinor:224 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/116df32b-c0fe-456f-bea6-2fdf33dc0039/volumes/kubernetes.io~projected/kube-api-access-plv7c DeviceMajor:0 DeviceMinor:212 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f338bc5c-18e1-4e62-be2d-93de38d4ae9d/volumes/kubernetes.io~projected/kube-api-access-x65pj DeviceMajor:0 DeviceMinor:201 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/616e70f1a66a6f876a1ba23e7a895523dc3c488c3dc17c2a36e0d2d5e260fbe5/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ed98cbbd-063e-43ed-bddc-ba216fbf56bb/volumes/kubernetes.io~projected/kube-api-access-w9hhh DeviceMajor:0 DeviceMinor:49 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d0156f0bb202ee1e9a13702d1b9662569f6daa6596a371330024765aa21c1d9/userdata/shm DeviceMajor:0 DeviceMinor:53 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/256e47eee838717b2a5550e39abbf29ab90699afeb6eb49bd5be65e316afe491/userdata/shm DeviceMajor:0 DeviceMinor:54 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/288087a8-c7ea-40a8-88bb-1b85be0aef55/volumes/kubernetes.io~projected/kube-api-access-tgf48 DeviceMajor:0 DeviceMinor:51 Capacity:31985721344 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:342b035e1591ec6 MacAddress:7a:03:a1:62:43:7f Speed:10000 Mtu:8901} {Name:6ba7ad2e2d6a4d8 MacAddress:66:8d:8c:d8:3f:ac Speed:10000 Mtu:8901} {Name:91eb40ca7bc8d9a MacAddress:e2:02:23:b2:be:66 Speed:10000 Mtu:8901} {Name:9625676fab8b03f MacAddress:4e:e2:7e:89:e4:e4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ea:58:22:c1 Speed:0 Mtu:9001} {Name:br-int MacAddress:02:01:18:e5:7f:60 Speed:0 Mtu:8901} {Name:d4fb16a264a1a0b MacAddress:2e:ae:e6:81:16:74 Speed:10000 Mtu:8901} {Name:d79bd8837dd89a2 MacAddress:ee:4f:01:74:cb:f4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ea:58:22:c1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:7a:77:00:86:fc:d9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fa:f5:ec:3a:1d:c7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164320768 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.230929 4542 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.230978 4542 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231245 4542 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231272 4542 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-206.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231384 4542 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231392 4542 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231402 4542 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231420 4542 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231511 4542 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231664 4542 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231776 4542 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231808 4542 kubelet.go:493] "Attempting to sync node with API server" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231820 4542 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231830 4542 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231839 4542 kubelet.go:405] "Adding apiserver pod source" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.231846 4542 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.232400 4542 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.232408 4542 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:59:43.233979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.232849 4542 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 12:59:43.234865 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.234852 4542 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 12:59:43.234865 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.234865 4542 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:59:43.234963 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.234953 4542 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 12:59:43.235131 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235121 4542 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 12:59:43.235131 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235149 4542 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 12:59:43.235241 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235232 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 12:59:43.235283 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235246 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 12:59:43.235283 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235252 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 12:59:43.235283 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235259 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 12:59:43.235283 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235264 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 12:59:43.235283 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235271 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 12:59:43.235283 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235276 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 12:59:43.235283 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235282 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 12:59:43.235563 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235289 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 12:59:43.235563 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235295 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 12:59:43.235563 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235309 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 12:59:43.235563 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235318 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 12:59:43.235563 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235342 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 12:59:43.235563 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.235348 4542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 12:59:43.238515 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.238499 4542 server.go:1263] "Started kubelet" Oct 31 12:59:43.238594 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.238536 4542 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 12:59:43.239093 ip-10-0-137-206 systemd[1]: Started Kubernetes Kubelet. Oct 31 12:59:43.239342 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.238576 4542 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 12:59:43.239916 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.239894 4542 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 12:59:43.240186 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.240169 4542 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 12:59:43.245041 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.244853 4542 server.go:317] "Adding debug handlers to kubelet server" Oct 31 12:59:43.246063 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.246040 4542 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 12:59:43.246145 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.246110 4542 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.246205 4542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:54:05 +0000 UTC" deadline="2028-05-16 23:11:09.401878525 +0000 UTC" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.246242 4542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13522h11m26.155640245s" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.246448 4542 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.246463 4542 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:43.246494 4542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-206.us-east-2.compute.internal\" not found" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.246532 4542 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248369 4542 factory.go:55] Registering systemd factory Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248384 4542 factory.go:223] Registration of the systemd container factory successfully Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248539 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1aaae2c3-0ea2-4061-a448-e50f82dcf823" volumeName="kubernetes.io/secret/1aaae2c3-0ea2-4061-a448-e50f82dcf823-agent-certs" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248605 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d525238-6ced-4f9f-89a6-56fb50fba909" volumeName="kubernetes.io/configmap/5d525238-6ced-4f9f-89a6-56fb50fba909-serviceca" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248624 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400" volumeName="kubernetes.io/empty-dir/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-data-volume" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248638 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e16e028-6e02-48e1-b54e-3517232c2d8b" volumeName="kubernetes.io/projected/9e16e028-6e02-48e1-b54e-3517232c2d8b-kube-api-access-872ln" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248657 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cabced46-3402-4172-aa1b-f056b9e01220" volumeName="kubernetes.io/projected/cabced46-3402-4172-aa1b-f056b9e01220-kube-api-access-xfpfl" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248671 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f338bc5c-18e1-4e62-be2d-93de38d4ae9d" volumeName="kubernetes.io/projected/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-kube-api-access-x65pj" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248690 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="116df32b-c0fe-456f-bea6-2fdf33dc0039" volumeName="kubernetes.io/projected/116df32b-c0fe-456f-bea6-2fdf33dc0039-kube-api-access-plv7c" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248711 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="791df070-f034-44c1-8098-df4065c48a32" volumeName="kubernetes.io/projected/791df070-f034-44c1-8098-df4065c48a32-kube-api-access-4dcr7" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248756 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="791df070-f034-44c1-8098-df4065c48a32" volumeName="kubernetes.io/secret/791df070-f034-44c1-8098-df4065c48a32-signing-key" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248770 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd465d2-5a1c-4fd4-b43f-a386fc814e3f" volumeName="kubernetes.io/projected/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-kube-api-access-ncmqn" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248782 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed98cbbd-063e-43ed-bddc-ba216fbf56bb" volumeName="kubernetes.io/projected/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-kube-api-access-w9hhh" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248801 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef470cac-0cd7-4fae-87e2-f6d431227d00" volumeName="kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248817 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9e011c-a51f-4738-b970-58ec77045b81" volumeName="kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-env-overrides" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248835 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef470cac-0cd7-4fae-87e2-f6d431227d00" volumeName="kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248856 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef470cac-0cd7-4fae-87e2-f6d431227d00" volumeName="kubernetes.io/projected/ef470cac-0cd7-4fae-87e2-f6d431227d00-kube-api-access-d7wgc" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248876 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9e011c-a51f-4738-b970-58ec77045b81" volumeName="kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-config" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248889 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400" volumeName="kubernetes.io/projected/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-api-access-xjnj2" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248901 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6266aa7-b443-49e9-88a0-987228617831" volumeName="kubernetes.io/projected/c6266aa7-b443-49e9-88a0-987228617831-kube-api-access-z4fm9" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248920 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9e011c-a51f-4738-b970-58ec77045b81" volumeName="kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-script-lib" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248935 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09700847-fa74-4444-9d56-5a4e45300c79" volumeName="kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248954 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6266aa7-b443-49e9-88a0-987228617831" volumeName="kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-service-ca-bundle" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248966 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6266aa7-b443-49e9-88a0-987228617831" volumeName="kubernetes.io/empty-dir/c6266aa7-b443-49e9-88a0-987228617831-archives-path" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.248979 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cabced46-3402-4172-aa1b-f056b9e01220" volumeName="kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-etc-tuned" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249003 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd465d2-5a1c-4fd4-b43f-a386fc814e3f" volumeName="kubernetes.io/configmap/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-iptables-alerter-script" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249018 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed98cbbd-063e-43ed-bddc-ba216fbf56bb" volumeName="kubernetes.io/empty-dir/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-tmp-dir" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249035 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef470cac-0cd7-4fae-87e2-f6d431227d00" volumeName="kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-binary-copy" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249049 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f338bc5c-18e1-4e62-be2d-93de38d4ae9d" volumeName="kubernetes.io/empty-dir/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-tmp-dir" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249063 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="288087a8-c7ea-40a8-88bb-1b85be0aef55" volumeName="kubernetes.io/projected/288087a8-c7ea-40a8-88bb-1b85be0aef55-kube-api-access-tgf48" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249090 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d525238-6ced-4f9f-89a6-56fb50fba909" volumeName="kubernetes.io/projected/5d525238-6ced-4f9f-89a6-56fb50fba909-kube-api-access-jtwpx" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249109 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83f70c7-ab6c-43b0-b86d-3d9d935973ec" volumeName="kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249131 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9e011c-a51f-4738-b970-58ec77045b81" volumeName="kubernetes.io/projected/fa9e011c-a51f-4738-b970-58ec77045b81-kube-api-access-svj5d" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249149 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9e011c-a51f-4738-b970-58ec77045b81" volumeName="kubernetes.io/secret/fa9e011c-a51f-4738-b970-58ec77045b81-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249163 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1aaae2c3-0ea2-4061-a448-e50f82dcf823" volumeName="kubernetes.io/configmap/1aaae2c3-0ea2-4061-a448-e50f82dcf823-konnectivity-ca" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249175 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="791df070-f034-44c1-8098-df4065c48a32" volumeName="kubernetes.io/configmap/791df070-f034-44c1-8098-df4065c48a32-signing-cabundle" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249194 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bf98082-9c6b-44e8-99e1-8ad8320b3483" volumeName="kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-daemon-config" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249208 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bf98082-9c6b-44e8-99e1-8ad8320b3483" volumeName="kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cni-binary-copy" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249226 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bf98082-9c6b-44e8-99e1-8ad8320b3483" volumeName="kubernetes.io/projected/7bf98082-9c6b-44e8-99e1-8ad8320b3483-kube-api-access-bw5pd" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249239 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400" volumeName="kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249252 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e16e028-6e02-48e1-b54e-3517232c2d8b" volumeName="kubernetes.io/secret/9e16e028-6e02-48e1-b54e-3517232c2d8b-cert" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249270 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cabced46-3402-4172-aa1b-f056b9e01220" volumeName="kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-tmp" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249284 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5a2f7b9-0ca0-4670-80cd-16e31aecb597" volumeName="kubernetes.io/projected/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kube-api-access-mll9t" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249301 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6266aa7-b443-49e9-88a0-987228617831" volumeName="kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-trusted-ca-bundle" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249314 4542 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f338bc5c-18e1-4e62-be2d-93de38d4ae9d" volumeName="kubernetes.io/configmap/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-config-volume" seLinuxMountContext="" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249326 4542 reconstruct.go:97] "Volume reconstruction finished" Oct 31 12:59:43.249416 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249334 4542 reconciler.go:29] "Reconciler: start to sync state" Oct 31 12:59:43.251217 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249713 4542 factory.go:153] Registering CRI-O factory Oct 31 12:59:43.251217 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249729 4542 factory.go:223] Registration of the crio container factory successfully Oct 31 12:59:43.251217 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249784 4542 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 12:59:43.251217 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249804 4542 factory.go:103] Registering Raw factory Oct 31 12:59:43.251217 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.249820 4542 manager.go:1199] Started watching for new ooms in manager Oct 31 12:59:43.251217 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.250040 4542 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:59:43.251217 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.251105 4542 manager.go:319] Starting recovery of all containers Oct 31 12:59:43.251808 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.251790 4542 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:59:43.252055 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.252041 4542 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:59:43.253096 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.252908 4542 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 12:59:43.273236 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.273210 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 12:59:43.273368 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.273336 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 12:59:43.281979 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.281951 4542 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 12:59:43.283707 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.283681 4542 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 12:59:43.283823 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.283712 4542 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 12:59:43.283823 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.283734 4542 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 12:59:43.283823 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:43.283774 4542 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 12:59:43.289042 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.289018 4542 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:59:43.291664 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.291636 4542 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="e4d6529470f55ac7954c3ce55cb53bccf1bcb35a82e4afc59420ee02c9b15195" exitCode=0 Oct 31 12:59:43.291664 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.291663 4542 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="4971798d01053b7f8621e22f6854e33973269734fda0fc08f42042850c46c364" exitCode=0 Oct 31 12:59:43.291817 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.291672 4542 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="e4e6d47662b7c502b5924c9874ba18d9d5c0ff53b9f8e2d7d42b5f779256ac48" exitCode=0 Oct 31 12:59:43.291817 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.291681 4542 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="903c8b7dc27e083a9aa00ca86f85226764d608e1936688b0e06585e9f5d04f38" exitCode=0 Oct 31 12:59:43.291817 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.291691 4542 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="72e2d4871ba42c56a68b202fdfbf6e51f119b004f3c7311fa1c397c87fe306fe" exitCode=0 Oct 31 12:59:43.291817 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.291699 4542 generic.go:356] "Generic (PLEG): container finished" podID="ef470cac-0cd7-4fae-87e2-f6d431227d00" containerID="0dd31abbf2a717c26a8854ee045466afdffc0113bd7ecac150f57f6703fe3d3d" exitCode=0 Oct 31 12:59:43.296162 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.296144 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 12:59:43.296581 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.296554 4542 generic.go:356] "Generic (PLEG): container finished" podID="fa9e011c-a51f-4738-b970-58ec77045b81" containerID="f6d377953c54ad445e341752d13ced406b97de99072a4304b6ab6f0f23f33263" exitCode=1 Oct 31 12:59:43.309272 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.309243 4542 generic.go:356] "Generic (PLEG): container finished" podID="7793224c37886f13184901ef3d534bb5" containerID="23b8079f1f1ece8c5d3772d16ed9b051482f80bf291fdc24b53e0d1413b40d0e" exitCode=0 Oct 31 12:59:43.323513 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.323493 4542 manager.go:324] Recovery completed Oct 31 12:59:43.337810 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.337760 4542 cpu_manager.go:226] "Starting" policy="none" Oct 31 12:59:43.337810 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.337772 4542 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 12:59:43.337810 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.337788 4542 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 12:59:43.338006 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.337903 4542 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 12:59:43.338349 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.337914 4542 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 12:59:43.338387 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.338351 4542 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 12:59:43.338387 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.338358 4542 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 12:59:43.338387 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.338364 4542 policy_none.go:50] "Start" Oct 31 12:59:43.338387 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.338374 4542 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 12:59:43.338387 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.338386 4542 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 12:59:43.338541 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.338531 4542 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 12:59:43.338571 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.338541 4542 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 12:59:43.338571 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.338547 4542 policy_none.go:44] "Start" Oct 31 12:59:43.342491 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.342449 4542 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 12:59:43.348380 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.348365 4542 manager.go:341] "Starting Device Plugin manager" Oct 31 12:59:43.348459 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:43.348388 4542 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 12:59:43.348459 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.348396 4542 server.go:87] "Starting device plugin registration server" Oct 31 12:59:43.348664 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.348651 4542 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 12:59:43.348730 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.348662 4542 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 12:59:43.348779 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.348735 4542 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 12:59:43.348893 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.348842 4542 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 12:59:43.348893 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.348879 4542 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 12:59:43.348893 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.348885 4542 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 12:59:43.349098 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.348985 4542 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:59:43.348868568Z","UUID":"fe765953-515b-4a29-9b24-231ad7d1b7e7","Handler":null,"Name":"","Endpoint":""} Oct 31 12:59:43.350224 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:43.350201 4542 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 12:59:43.350300 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.350256 4542 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 12:59:43.350300 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.350277 4542 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 12:59:43.384535 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.384501 4542 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal"] Oct 31 12:59:43.385023 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.384988 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" event={"ID":"7793224c37886f13184901ef3d534bb5","Type":"ContainerStarted","Data":"52ff25f9d1c11dcda28035fd35359ae8b116797c9d58cd6eaceeebb088c4279b"} Oct 31 12:59:43.385126 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.385030 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" event={"ID":"7793224c37886f13184901ef3d534bb5","Type":"ContainerDied","Data":"23b8079f1f1ece8c5d3772d16ed9b051482f80bf291fdc24b53e0d1413b40d0e"} Oct 31 12:59:43.385126 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.385047 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" event={"ID":"7793224c37886f13184901ef3d534bb5","Type":"ContainerStarted","Data":"256e47eee838717b2a5550e39abbf29ab90699afeb6eb49bd5be65e316afe491"} Oct 31 12:59:43.385126 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.385061 4542 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.385213 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.385068 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" event={"ID":"df9f494cd77ee5cda911dbf4d5fd20aa","Type":"ContainerStarted","Data":"17e8aefe897f0ca8e115165c8a997434823ceb2c4dfe6a26f287f4fa359134a1"} Oct 31 12:59:43.385213 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.385150 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" event={"ID":"df9f494cd77ee5cda911dbf4d5fd20aa","Type":"ContainerStarted","Data":"7d0156f0bb202ee1e9a13702d1b9662569f6daa6596a371330024765aa21c1d9"} Oct 31 12:59:43.385213 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.385177 4542 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.397813 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.397796 4542 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:59:43.397866 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:43.397833 4542 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.398785 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.398776 4542 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:59:43.398819 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:43.398797 4542 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.449083 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.449058 4542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:59:43.450196 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.450182 4542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:59:43.450281 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.450208 4542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:59:43.450281 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.450218 4542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-206.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:59:43.450281 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.450259 4542 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.453725 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.453704 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.453797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.453738 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.453797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.453787 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df9f494cd77ee5cda911dbf4d5fd20aa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"df9f494cd77ee5cda911dbf4d5fd20aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.464931 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.464917 4542 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.464993 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.464961 4542 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.554186 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.554157 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.554263 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.554197 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.554263 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.554222 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.554263 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.554224 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df9f494cd77ee5cda911dbf4d5fd20aa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"df9f494cd77ee5cda911dbf4d5fd20aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.554385 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.554259 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df9f494cd77ee5cda911dbf4d5fd20aa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"df9f494cd77ee5cda911dbf4d5fd20aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:43.554385 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:43.554281 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7793224c37886f13184901ef3d534bb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" (UID: \"7793224c37886f13184901ef3d534bb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:44.232900 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.232864 4542 apiserver.go:52] "Watching apiserver" Oct 31 12:59:44.245713 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.245681 4542 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 12:59:44.246171 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.246145 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c2knk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-w58sk","kube-system/konnectivity-agent-rwtn6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw","openshift-dns/node-resolver-cs9pm","openshift-image-registry/node-ca-s7fjc","openshift-network-diagnostics/network-check-target-sjsfh","openshift-service-ca/service-ca-5fffc97bdf-6c5t9","openshift-multus/multus-vgm78","openshift-cluster-node-tuning-operator/tuned-gvlv8","openshift-multus/multus-additional-cni-plugins-wprss","openshift-multus/network-metrics-daemon-sq2q6","openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr","openshift-ingress-canary/ingress-canary-rm7b9","openshift-dns/dns-default-ltf7k","openshift-network-operator/iptables-alerter-l2brm","openshift-insights/periodic-gathering-q8jf6-vd797","kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-pbk8s"] Oct 31 12:59:44.249306 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.249282 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:44.250909 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.250888 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:44.251636 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.251612 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.251734 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.251674 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.251734 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.251710 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 12:59:44.252588 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.252567 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.252686 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.252666 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.252736 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.252686 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.252789 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.252748 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 12:59:44.252841 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.252792 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.252906 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.252882 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.253087 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.253072 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.253612 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.253593 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.254434 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.254416 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.254517 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.254506 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.254570 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.254553 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.255081 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.255066 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 12:59:44.255247 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.255231 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 12:59:44.255558 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.255541 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2x4wq\"" Oct 31 12:59:44.255631 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.255614 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.256113 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.256089 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5b9kz\"" Oct 31 12:59:44.256199 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.256164 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.256521 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.256502 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lt4jt\"" Oct 31 12:59:44.257011 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.256991 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-etc-tuned\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.257122 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257035 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vtxsh\"" Oct 31 12:59:44.257337 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257319 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.257405 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257356 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 12:59:44.257405 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257361 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g22bl\"" Oct 31 12:59:44.257405 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257389 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.257559 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257038 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 12:59:44.257604 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257035 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpfl\" (UniqueName: \"kubernetes.io/projected/cabced46-3402-4172-aa1b-f056b9e01220-kube-api-access-xfpfl\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.257654 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257627 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drb8j\"" Oct 31 12:59:44.257699 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257670 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-etc-tuned\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.257766 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257750 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-run\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.257821 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257780 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-sys\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.257869 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257817 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-lib-modules\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.257917 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257894 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 12:59:44.257961 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257849 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-dbus\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:44.257961 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257944 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 12:59:44.257961 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257950 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:44.258104 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257984 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.258104 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258009 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-kubelet-config\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:44.258104 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258083 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.258571 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258100 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-modprobe-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.258571 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258165 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysconfig\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.258571 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257919 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 12:59:44.258571 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258205 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-kubernetes\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.258571 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258283 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 12:59:44.258571 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258344 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-conf\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.258571 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258385 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-systemd\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258572 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-var-lib-kubelet\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258574 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258599 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.257986 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hwtqn\"" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258183 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mw6wr\"" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258663 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258679 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2xlcn\"" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258714 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2kvxk\"" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258710 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-host\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258807 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-tmp\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.258903 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258824 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.259352 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258952 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cabced46-3402-4172-aa1b-f056b9e01220-tmp\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.259352 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258964 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:59:44.259352 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.258991 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gth9v\"" Oct 31 12:59:44.259352 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259066 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lsjfn\"" Oct 31 12:59:44.259352 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259130 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.259352 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259142 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7lhn5\"" Oct 31 12:59:44.259352 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259235 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:59:44.259352 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259308 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 12:59:44.259722 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259707 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 12:59:44.259772 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259758 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.259830 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259806 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftfm5\"" Oct 31 12:59:44.259940 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259925 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 12:59:44.259995 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259934 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.259995 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259924 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lmj6s\"" Oct 31 12:59:44.259995 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259928 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.259995 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.259958 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lt77s\"" Oct 31 12:59:44.260132 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.260011 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.260218 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.260201 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 12:59:44.265178 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.265163 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 12:59:44.265565 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.265546 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 12:59:44.265658 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.265545 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 12:59:44.265658 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.265577 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 12:59:44.265658 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.265603 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 12:59:44.265974 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.265940 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 12:59:44.266594 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.266578 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qpv6m\"" Oct 31 12:59:44.266796 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.266783 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jl9zz\"" Oct 31 12:59:44.266839 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.266801 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 12:59:44.268440 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.268425 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpfl\" (UniqueName: \"kubernetes.io/projected/cabced46-3402-4172-aa1b-f056b9e01220-kube-api-access-xfpfl\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.269413 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.269396 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-original-pull-secret\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:44.326346 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.326328 4542 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:44.326522 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.326508 4542 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:44.339381 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.339361 4542 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:59:44.339447 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:44.339396 4542 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:44.340464 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.340452 4542 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:59:44.340513 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:44.340495 4542 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-206.us-east-2.compute.internal" Oct 31 12:59:44.348437 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.348422 4542 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 12:59:44.359568 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359546 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-config-volume\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:44.359643 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359573 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgf48\" (UniqueName: \"kubernetes.io/projected/288087a8-c7ea-40a8-88bb-1b85be0aef55-kube-api-access-tgf48\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:44.359643 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359601 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-script-lib\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.359643 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359618 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-sys-fs\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.359643 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359633 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65pj\" (UniqueName: \"kubernetes.io/projected/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-kube-api-access-x65pj\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:44.359807 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359647 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aaae2c3-0ea2-4061-a448-e50f82dcf823-konnectivity-ca\") pod \"konnectivity-agent-rwtn6\" (UID: \"1aaae2c3-0ea2-4061-a448-e50f82dcf823\") " pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:44.359807 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359750 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-crio-socket\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.359897 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359879 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:44.359948 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359910 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-script-lib\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.360003 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359944 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-dbus\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:44.360003 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.359975 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-systemd-units\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.360085 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360014 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-systemd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.360085 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360022 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-config-volume\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:44.360085 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360033 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aaae2c3-0ea2-4061-a448-e50f82dcf823-konnectivity-ca\") pod \"konnectivity-agent-rwtn6\" (UID: \"1aaae2c3-0ea2-4061-a448-e50f82dcf823\") " pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:44.360085 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360022 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-dbus\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:44.360085 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360047 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.360085 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360078 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-env-overrides\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.360293 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360113 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872ln\" (UniqueName: \"kubernetes.io/projected/9e16e028-6e02-48e1-b54e-3517232c2d8b-kube-api-access-872ln\") pod \"ingress-canary-rm7b9\" (UID: \"9e16e028-6e02-48e1-b54e-3517232c2d8b\") " pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:44.360293 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360175 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6266aa7-b443-49e9-88a0-987228617831-archives-path\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:44.360293 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360194 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwpx\" (UniqueName: \"kubernetes.io/projected/5d525238-6ced-4f9f-89a6-56fb50fba909-kube-api-access-jtwpx\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:44.360293 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360215 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-env-overrides\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.360293 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360246 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cnibin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.360293 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360269 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cni-binary-copy\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.360293 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360281 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6266aa7-b443-49e9-88a0-987228617831-archives-path\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:44.360293 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360284 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-bin\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.360635 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360321 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-netd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.360635 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360362 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-tmp-dir\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:44.360635 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360403 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9hhh\" (UniqueName: \"kubernetes.io/projected/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-kube-api-access-w9hhh\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:44.360635 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360436 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-iptables-alerter-script\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:44.360635 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360457 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-tmp-dir\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:44.360635 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360461 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-system-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.360635 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360515 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cni-binary-copy\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.360635 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360527 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.360635 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360566 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plv7c\" (UniqueName: \"kubernetes.io/projected/116df32b-c0fe-456f-bea6-2fdf33dc0039-kube-api-access-plv7c\") pod \"migrator-66df5d496b-lt9fr\" (UID: \"116df32b-c0fe-456f-bea6-2fdf33dc0039\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr" Oct 31 12:59:44.360635 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360599 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcr7\" (UniqueName: \"kubernetes.io/projected/791df070-f034-44c1-8098-df4065c48a32-kube-api-access-4dcr7\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:44.361040 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360714 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d525238-6ced-4f9f-89a6-56fb50fba909-host\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:44.361040 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360733 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.361040 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360748 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-etc-selinux\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.361040 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360760 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-iptables-alerter-script\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:44.361040 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360812 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.361040 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360871 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/791df070-f034-44c1-8098-df4065c48a32-signing-key\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:44.361040 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360899 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-run\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.361040 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.360956 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-run\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.361040 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361010 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-sys\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361051 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361057 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-sys\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361081 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svj5d\" (UniqueName: \"kubernetes.io/projected/fa9e011c-a51f-4738-b970-58ec77045b81-kube-api-access-svj5d\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361106 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361107 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/791df070-f034-44c1-8098-df4065c48a32-signing-key\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361146 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-kubelet-config\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361173 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw5pd\" (UniqueName: \"kubernetes.io/projected/7bf98082-9c6b-44e8-99e1-8ad8320b3483-kube-api-access-bw5pd\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361221 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361225 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wgc\" (UniqueName: \"kubernetes.io/projected/ef470cac-0cd7-4fae-87e2-f6d431227d00-kube-api-access-d7wgc\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361252 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmqn\" (UniqueName: \"kubernetes.io/projected/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-kube-api-access-ncmqn\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361269 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361282 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-modprobe-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361305 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysconfig\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361222 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d83f70c7-ab6c-43b0-b86d-3d9d935973ec-kubelet-config\") pod \"global-pull-secret-syncer-w58sk\" (UID: \"d83f70c7-ab6c-43b0-b86d-3d9d935973ec\") " pod="kube-system/global-pull-secret-syncer-w58sk" Oct 31 12:59:44.361389 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361356 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysconfig\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361459 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-conf\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361520 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d525238-6ced-4f9f-89a6-56fb50fba909-serviceca\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361463 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-modprobe-d\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361552 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa9e011c-a51f-4738-b970-58ec77045b81-ovn-node-metrics-cert\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361643 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mll9t\" (UniqueName: \"kubernetes.io/projected/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kube-api-access-mll9t\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361659 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361561 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-sysctl-conf\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361677 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa9e011c-a51f-4738-b970-58ec77045b81-ovn-node-metrics-cert\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361682 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-k8s-cni-cncf-io\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361733 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-multus-certs\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361816 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-etc-kubernetes\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361842 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-kubelet\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361885 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-var-lib-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361926 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-cnibin\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361956 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aaae2c3-0ea2-4061-a448-e50f82dcf823-agent-certs\") pod \"konnectivity-agent-rwtn6\" (UID: \"1aaae2c3-0ea2-4061-a448-e50f82dcf823\") " pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.361991 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-data-volume\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362031 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-config\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362081 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-service-ca-bundle\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:44.362101 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362109 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-socket-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362102 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-data-volume\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362149 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-netns\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362177 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-trusted-ca-bundle\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362232 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d525238-6ced-4f9f-89a6-56fb50fba909-serviceca\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362263 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-hosts-file\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362292 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnj2\" (UniqueName: \"kubernetes.io/projected/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-api-access-xjnj2\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362319 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-kubernetes\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362346 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-var-lib-kubelet\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362369 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-host\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362420 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aaae2c3-0ea2-4061-a448-e50f82dcf823-agent-certs\") pod \"konnectivity-agent-rwtn6\" (UID: \"1aaae2c3-0ea2-4061-a448-e50f82dcf823\") " pod="kube-system/konnectivity-agent-rwtn6" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362439 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-bin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362497 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-kubelet\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362497 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-host\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362544 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-var-lib-kubelet\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362570 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-hostroot\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362597 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-conf-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362633 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-daemon-config\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362758 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa9e011c-a51f-4738-b970-58ec77045b81-ovnkube-config\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362757 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-ovn\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362797 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e16e028-6e02-48e1-b54e-3517232c2d8b-cert\") pod \"ingress-canary-rm7b9\" (UID: \"9e16e028-6e02-48e1-b54e-3517232c2d8b\") " pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362810 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-service-ca-bundle\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362827 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4fm9\" (UniqueName: \"kubernetes.io/projected/c6266aa7-b443-49e9-88a0-987228617831-kube-api-access-z4fm9\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362798 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-kubernetes\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362813 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-trusted-ca-bundle\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362907 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-registration-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362955 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.362977 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362993 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-daemon-config\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.362998 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-tmp-dir\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363057 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363058 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e16e028-6e02-48e1-b54e-3517232c2d8b-cert\") pod \"ingress-canary-rm7b9\" (UID: \"9e16e028-6e02-48e1-b54e-3517232c2d8b\") " pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363112 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-tmp-dir\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363162 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-lib-modules\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363165 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363227 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-netns\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363248 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-lib-modules\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363256 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-etc-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363278 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-node-log\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363335 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363364 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363389 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-multus\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363486 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-slash\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363503 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-log-socket\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363525 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-system-cni-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363556 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-binary-copy\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363586 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363717 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363723 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363745 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/791df070-f034-44c1-8098-df4065c48a32-signing-cabundle\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363751 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef470cac-0cd7-4fae-87e2-f6d431227d00-cni-binary-copy\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363770 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-systemd\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363795 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-socket-dir-parent\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363803 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cabced46-3402-4172-aa1b-f056b9e01220-etc-systemd\") pod \"tuned-gvlv8\" (UID: \"cabced46-3402-4172-aa1b-f056b9e01220\") " pod="openshift-cluster-node-tuning-operator/tuned-gvlv8" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363814 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-device-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363836 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-host-slash\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:44.363986 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.363978 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/791df070-f034-44c1-8098-df4065c48a32-signing-cabundle\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:44.371081 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.371061 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65pj\" (UniqueName: \"kubernetes.io/projected/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-kube-api-access-x65pj\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:44.371155 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.371090 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgf48\" (UniqueName: \"kubernetes.io/projected/288087a8-c7ea-40a8-88bb-1b85be0aef55-kube-api-access-tgf48\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:44.371412 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.371395 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9hhh\" (UniqueName: \"kubernetes.io/projected/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-kube-api-access-w9hhh\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:44.371750 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.371725 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872ln\" (UniqueName: \"kubernetes.io/projected/9e16e028-6e02-48e1-b54e-3517232c2d8b-kube-api-access-872ln\") pod \"ingress-canary-rm7b9\" (UID: \"9e16e028-6e02-48e1-b54e-3517232c2d8b\") " pod="openshift-ingress-canary/ingress-canary-rm7b9" Oct 31 12:59:44.371865 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.371830 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmqn\" (UniqueName: \"kubernetes.io/projected/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-kube-api-access-ncmqn\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:44.371971 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.371928 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwpx\" (UniqueName: \"kubernetes.io/projected/5d525238-6ced-4f9f-89a6-56fb50fba909-kube-api-access-jtwpx\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:44.372025 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.371994 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcr7\" (UniqueName: \"kubernetes.io/projected/791df070-f034-44c1-8098-df4065c48a32-kube-api-access-4dcr7\") pod \"service-ca-5fffc97bdf-6c5t9\" (UID: \"791df070-f034-44c1-8098-df4065c48a32\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6c5t9" Oct 31 12:59:44.372166 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.372142 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plv7c\" (UniqueName: \"kubernetes.io/projected/116df32b-c0fe-456f-bea6-2fdf33dc0039-kube-api-access-plv7c\") pod \"migrator-66df5d496b-lt9fr\" (UID: \"116df32b-c0fe-456f-bea6-2fdf33dc0039\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lt9fr" Oct 31 12:59:44.372269 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.372255 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw5pd\" (UniqueName: \"kubernetes.io/projected/7bf98082-9c6b-44e8-99e1-8ad8320b3483-kube-api-access-bw5pd\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.372306 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.372276 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svj5d\" (UniqueName: \"kubernetes.io/projected/fa9e011c-a51f-4738-b970-58ec77045b81-kube-api-access-svj5d\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.372564 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.372523 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mll9t\" (UniqueName: \"kubernetes.io/projected/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kube-api-access-mll9t\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.373136 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.373121 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wgc\" (UniqueName: \"kubernetes.io/projected/ef470cac-0cd7-4fae-87e2-f6d431227d00-kube-api-access-d7wgc\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.373508 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.373487 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4fm9\" (UniqueName: \"kubernetes.io/projected/c6266aa7-b443-49e9-88a0-987228617831-kube-api-access-z4fm9\") pod \"periodic-gathering-q8jf6-vd797\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 12:59:44.376177 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.376162 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8j2z\" (UniqueName: \"kubernetes.io/projected/09700847-fa74-4444-9d56-5a4e45300c79-kube-api-access-r8j2z\") pod \"network-check-target-sjsfh\" (UID: \"09700847-fa74-4444-9d56-5a4e45300c79\") " pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:44.376223 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.376209 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnj2\" (UniqueName: \"kubernetes.io/projected/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-kube-api-access-xjnj2\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.464631 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464605 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-systemd-units\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464631 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464632 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-systemd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464647 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464664 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cnibin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464686 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-bin\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464699 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-netd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464714 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-system-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464715 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-systemd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464729 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464716 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-systemd-units\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464750 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d525238-6ced-4f9f-89a6-56fb50fba909-host\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464754 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-bin\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464768 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-cni-netd\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464769 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464769 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-system-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464775 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.464797 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464761 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-cnibin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464816 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464817 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-cni-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464849 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d525238-6ced-4f9f-89a6-56fb50fba909-host\") pod \"node-ca-s7fjc\" (UID: \"5d525238-6ced-4f9f-89a6-56fb50fba909\") " pod="openshift-image-registry/node-ca-s7fjc" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464858 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-etc-selinux\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464885 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-etc-selinux\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464891 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464941 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.464958 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465002 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-k8s-cni-cncf-io\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465027 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-multus-certs\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465049 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-k8s-cni-cncf-io\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465050 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-etc-kubernetes\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465073 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-kubelet\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465085 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-etc-kubernetes\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465029 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465087 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-var-lib-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465106 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-var-lib-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465119 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-kubelet\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465130 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-cnibin\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465157 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-socket-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465134 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-multus-certs\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465182 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-netns\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465194 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-cnibin\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465205 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-hosts-file\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465211 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-socket-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465226 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-run-netns\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465233 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-bin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465255 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-kubelet\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465279 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-hostroot\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465302 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-conf-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465282 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed98cbbd-063e-43ed-bddc-ba216fbf56bb-hosts-file\") pod \"node-resolver-cs9pm\" (UID: \"ed98cbbd-063e-43ed-bddc-ba216fbf56bb\") " pod="openshift-dns/node-resolver-cs9pm" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465310 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-kubelet\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465305 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-bin\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465326 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-hostroot\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465327 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-ovn\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465340 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-conf-dir\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465359 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-registration-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465365 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-run-ovn\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465378 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465398 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-netns\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465416 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-registration-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465420 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-etc-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465448 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-node-log\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465451 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-etc-openvswitch\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465460 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-node-log\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465460 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-run-netns\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465498 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.465462 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465512 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465522 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-multus\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465554 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-slash\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465577 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-log-socket\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465585 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-host-var-lib-cni-multus\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465600 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-system-cni-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465620 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-host-slash\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465625 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465633 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa9e011c-a51f-4738-b970-58ec77045b81-log-socket\") pod \"ovnkube-node-pbk8s\" (UID: \"fa9e011c-a51f-4738-b970-58ec77045b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465648 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef470cac-0cd7-4fae-87e2-f6d431227d00-system-cni-dir\") pod \"multus-additional-cni-plugins-wprss\" (UID: \"ef470cac-0cd7-4fae-87e2-f6d431227d00\") " pod="openshift-multus/multus-additional-cni-plugins-wprss" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465649 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-socket-dir-parent\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465682 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-device-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465698 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-host-slash\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465698 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bf98082-9c6b-44e8-99e1-8ad8320b3483-multus-socket-dir-parent\") pod \"multus-vgm78\" (UID: \"7bf98082-9c6b-44e8-99e1-8ad8320b3483\") " pod="openshift-multus/multus-vgm78" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465714 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-sys-fs\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465740 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-crio-socket\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:44.465725 4542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465750 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecd465d2-5a1c-4fd4-b43f-a386fc814e3f-host-slash\") pod \"iptables-alerter-l2brm\" (UID: \"ecd465d2-5a1c-4fd4-b43f-a386fc814e3f\") " pod="openshift-network-operator/iptables-alerter-l2brm" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465756 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465754 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-device-dir\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465771 4542 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465802 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-crio-socket\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.465802 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5a2f7b9-0ca0-4670-80cd-16e31aecb597-sys-fs\") pod \"aws-ebs-csi-driver-node-mblqw\" (UID: \"a5a2f7b9-0ca0-4670-80cd-16e31aecb597\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mblqw" Oct 31 12:59:44.466951 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:44.465813 4542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls podName:f338bc5c-18e1-4e62-be2d-93de38d4ae9d nodeName:}" failed. No retries permitted until 2026-10-31 12:59:44.965793466 +0000 UTC m=+1.810959883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls") pod "dns-default-ltf7k" (UID: "f338bc5c-18e1-4e62-be2d-93de38d4ae9d") : secret "dns-default-metrics-tls" not found Oct 31 12:59:44.467921 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.467905 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2knk\" (UID: \"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400\") " pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.468105 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.468086 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288087a8-c7ea-40a8-88bb-1b85be0aef55-metrics-certs\") pod \"network-metrics-daemon-sq2q6\" (UID: \"288087a8-c7ea-40a8-88bb-1b85be0aef55\") " pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:44.550157 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.550113 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sq2q6" Oct 31 12:59:44.556531 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.556515 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2knk" Oct 31 12:59:44.656527 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.656035 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s7fjc_5d525238-6ced-4f9f-89a6-56fb50fba909/node-ca/0.log" Oct 31 12:59:44.854405 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.854381 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sq2q6"] Oct 31 12:59:44.857781 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:44.857739 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288087a8_c7ea_40a8_88bb_1b85be0aef55.slice/crio-1555ad50230e4385ef9cbeb9da0280183299751c7fb5610fd0bbdfb5ae2f4e1a WatchSource:0}: Error finding container 1555ad50230e4385ef9cbeb9da0280183299751c7fb5610fd0bbdfb5ae2f4e1a: Status 404 returned error can't find the container with id 1555ad50230e4385ef9cbeb9da0280183299751c7fb5610fd0bbdfb5ae2f4e1a Oct 31 12:59:44.859951 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.859927 4542 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:59:44.885970 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.885951 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2knk"] Oct 31 12:59:44.889588 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:44.889567 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2cb3ac_95ac_4d5b_96ac_cd6e2c382400.slice/crio-b130f293053a92a51a06a8bbf070e49fc79eb488b25d1e441a51e7db1e6ae998 WatchSource:0}: Error finding container b130f293053a92a51a06a8bbf070e49fc79eb488b25d1e441a51e7db1e6ae998: Status 404 returned error can't find the container with id b130f293053a92a51a06a8bbf070e49fc79eb488b25d1e441a51e7db1e6ae998 Oct 31 12:59:44.969564 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:44.969542 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:44.969716 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:44.969672 4542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:59:44.969751 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:44.969724 4542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls podName:f338bc5c-18e1-4e62-be2d-93de38d4ae9d nodeName:}" failed. No retries permitted until 2026-10-31 12:59:45.969711654 +0000 UTC m=+2.814878055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls") pod "dns-default-ltf7k" (UID: "f338bc5c-18e1-4e62-be2d-93de38d4ae9d") : secret "dns-default-metrics-tls" not found Oct 31 12:59:45.147372 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:45.147308 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 12:59:45.256583 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:45.256560 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rm7b9_9e16e028-6e02-48e1-b54e-3517232c2d8b/serve-healthcheck-canary/0.log" Oct 31 12:59:45.331621 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:45.331591 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2knk" event={"ID":"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400","Type":"ContainerStarted","Data":"e549a95bf00e84a536acdadb0beedea82a4153ba0649e188c007fbb8b7172aa5"} Oct 31 12:59:45.331764 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:45.331656 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2knk" event={"ID":"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400","Type":"ContainerStarted","Data":"b130f293053a92a51a06a8bbf070e49fc79eb488b25d1e441a51e7db1e6ae998"} Oct 31 12:59:45.332818 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:45.332791 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sq2q6" event={"ID":"288087a8-c7ea-40a8-88bb-1b85be0aef55","Type":"ContainerStarted","Data":"1555ad50230e4385ef9cbeb9da0280183299751c7fb5610fd0bbdfb5ae2f4e1a"} Oct 31 12:59:45.455366 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:45.455261 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lt9fr_116df32b-c0fe-456f-bea6-2fdf33dc0039/migrator/0.log" Oct 31 12:59:45.539552 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:45.539524 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:45.895607 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:45.895578 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:45.978516 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:45.978467 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:45.978647 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:45.978569 4542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:59:45.978647 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:45.978636 4542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls podName:f338bc5c-18e1-4e62-be2d-93de38d4ae9d nodeName:}" failed. No retries permitted until 2026-10-31 12:59:47.978621222 +0000 UTC m=+4.823787617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls") pod "dns-default-ltf7k" (UID: "f338bc5c-18e1-4e62-be2d-93de38d4ae9d") : secret "dns-default-metrics-tls" not found Oct 31 12:59:46.336407 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:46.336384 4542 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:59:47.342354 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:47.342314 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2knk" event={"ID":"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400","Type":"ContainerStarted","Data":"74b36125281b5049e9e01e8e0b94732c165eb35ff15a282f19924c50b8f453af"} Oct 31 12:59:47.344559 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:47.344528 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sq2q6" event={"ID":"288087a8-c7ea-40a8-88bb-1b85be0aef55","Type":"ContainerStarted","Data":"1bb562b7766a5440829655ce266bbd05bf3147ff57447036907f672d901be5bb"} Oct 31 12:59:47.344673 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:47.344581 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sq2q6" event={"ID":"288087a8-c7ea-40a8-88bb-1b85be0aef55","Type":"ContainerStarted","Data":"3ba7d98a57dcff06a2eaaa818f7fd72f89bd32bb5d2d3af895ad45742ac3ef24"} Oct 31 12:59:47.991934 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:47.991904 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:47.992048 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:47.992022 4542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:59:47.992091 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:47.992066 4542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls podName:f338bc5c-18e1-4e62-be2d-93de38d4ae9d nodeName:}" failed. No retries permitted until 2026-10-31 12:59:51.992053359 +0000 UTC m=+8.837219754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls") pod "dns-default-ltf7k" (UID: "f338bc5c-18e1-4e62-be2d-93de38d4ae9d") : secret "dns-default-metrics-tls" not found Oct 31 12:59:48.350064 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:48.350021 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2knk" event={"ID":"7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400","Type":"ContainerStarted","Data":"0878cc008536b0f68fc45c31a183cd91e75aeeeee67a86008fede52884ed3cdb"} Oct 31 12:59:52.016985 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:52.016946 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:52.019270 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:52.019247 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f338bc5c-18e1-4e62-be2d-93de38d4ae9d-metrics-tls\") pod \"dns-default-ltf7k\" (UID: \"f338bc5c-18e1-4e62-be2d-93de38d4ae9d\") " pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:52.052105 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:52.052081 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:52.322077 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:52.322050 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ltf7k"] Oct 31 12:59:52.325035 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:52.325005 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf338bc5c_18e1_4e62_be2d_93de38d4ae9d.slice/crio-01908e62114833870683e077f13639750e04cb92f3295591114c21e483dc3108 WatchSource:0}: Error finding container 01908e62114833870683e077f13639750e04cb92f3295591114c21e483dc3108: Status 404 returned error can't find the container with id 01908e62114833870683e077f13639750e04cb92f3295591114c21e483dc3108 Oct 31 12:59:52.367103 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:52.367064 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ltf7k" event={"ID":"f338bc5c-18e1-4e62-be2d-93de38d4ae9d","Type":"ContainerStarted","Data":"01908e62114833870683e077f13639750e04cb92f3295591114c21e483dc3108"} Oct 31 12:59:53.745618 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:53.745592 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbk8s" Oct 31 12:59:53.834700 ip-10-0-137-206 kubenswrapper[4542]: E1031 12:59:53.834659 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 12:59:54.373987 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:54.373953 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ltf7k" event={"ID":"f338bc5c-18e1-4e62-be2d-93de38d4ae9d","Type":"ContainerStarted","Data":"4dac70edc41b768006683ae0d3df58331604542b20447002f43a1e563a2bdcb7"} Oct 31 12:59:54.374116 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:54.374034 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ltf7k" event={"ID":"f338bc5c-18e1-4e62-be2d-93de38d4ae9d","Type":"ContainerStarted","Data":"e53c875935ac8058e862438dfaebaf96ea480a582784bbe2335a0e2bad6e900f"} Oct 31 12:59:54.374116 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:54.374093 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ltf7k" Oct 31 12:59:57.540885 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.540852 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-79n4w"] Oct 31 12:59:57.544219 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.544195 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-79n4w" Oct 31 12:59:57.550831 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.550798 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-jdgvc\"" Oct 31 12:59:57.560006 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.559979 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 12:59:57.560139 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.560092 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 12:59:57.571851 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.571829 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-79n4w"] Oct 31 12:59:57.641160 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.641120 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84f677497d-dp9bl"] Oct 31 12:59:57.648085 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.648058 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.658339 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.658313 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f7dx\" (UniqueName: \"kubernetes.io/projected/cd02e70d-668b-42d9-928e-ca8ddfd352d0-kube-api-access-2f7dx\") pod \"downloads-6dd789c8b7-79n4w\" (UID: \"cd02e70d-668b-42d9-928e-ca8ddfd352d0\") " pod="openshift-console/downloads-6dd789c8b7-79n4w" Oct 31 12:59:57.658472 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.658354 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd02e70d-668b-42d9-928e-ca8ddfd352d0-tmp\") pod \"downloads-6dd789c8b7-79n4w\" (UID: \"cd02e70d-668b-42d9-928e-ca8ddfd352d0\") " pod="openshift-console/downloads-6dd789c8b7-79n4w" Oct 31 12:59:57.660235 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.660214 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-64mjv\"" Oct 31 12:59:57.660336 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.660322 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 12:59:57.663508 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.663489 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 12:59:57.663596 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.663528 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 12:59:57.663596 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.663552 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 12:59:57.667136 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.667119 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f677497d-dp9bl"] Oct 31 12:59:57.759384 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759356 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09ac04c7-a98d-422d-8f55-3a1c4a41d952-installation-pull-secrets\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.759626 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759408 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd02e70d-668b-42d9-928e-ca8ddfd352d0-tmp\") pod \"downloads-6dd789c8b7-79n4w\" (UID: \"cd02e70d-668b-42d9-928e-ca8ddfd352d0\") " pod="openshift-console/downloads-6dd789c8b7-79n4w" Oct 31 12:59:57.759626 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759496 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q6sn\" (UniqueName: \"kubernetes.io/projected/09ac04c7-a98d-422d-8f55-3a1c4a41d952-kube-api-access-8q6sn\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.759626 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759536 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09ac04c7-a98d-422d-8f55-3a1c4a41d952-registry-tls\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.759626 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759553 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09ac04c7-a98d-422d-8f55-3a1c4a41d952-bound-sa-token\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.759626 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759569 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09ac04c7-a98d-422d-8f55-3a1c4a41d952-ca-trust-extracted\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.759853 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759692 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09ac04c7-a98d-422d-8f55-3a1c4a41d952-image-registry-private-configuration\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.759853 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759736 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09ac04c7-a98d-422d-8f55-3a1c4a41d952-trusted-ca\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.759853 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759791 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f7dx\" (UniqueName: \"kubernetes.io/projected/cd02e70d-668b-42d9-928e-ca8ddfd352d0-kube-api-access-2f7dx\") pod \"downloads-6dd789c8b7-79n4w\" (UID: \"cd02e70d-668b-42d9-928e-ca8ddfd352d0\") " pod="openshift-console/downloads-6dd789c8b7-79n4w" Oct 31 12:59:57.759853 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759795 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd02e70d-668b-42d9-928e-ca8ddfd352d0-tmp\") pod \"downloads-6dd789c8b7-79n4w\" (UID: \"cd02e70d-668b-42d9-928e-ca8ddfd352d0\") " pod="openshift-console/downloads-6dd789c8b7-79n4w" Oct 31 12:59:57.760018 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.759901 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09ac04c7-a98d-422d-8f55-3a1c4a41d952-registry-certificates\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.774652 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.774629 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f7dx\" (UniqueName: \"kubernetes.io/projected/cd02e70d-668b-42d9-928e-ca8ddfd352d0-kube-api-access-2f7dx\") pod \"downloads-6dd789c8b7-79n4w\" (UID: \"cd02e70d-668b-42d9-928e-ca8ddfd352d0\") " pod="openshift-console/downloads-6dd789c8b7-79n4w" Oct 31 12:59:57.856269 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.856239 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-79n4w" Oct 31 12:59:57.860448 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.860427 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q6sn\" (UniqueName: \"kubernetes.io/projected/09ac04c7-a98d-422d-8f55-3a1c4a41d952-kube-api-access-8q6sn\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.860576 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.860462 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09ac04c7-a98d-422d-8f55-3a1c4a41d952-registry-tls\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.860576 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.860504 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09ac04c7-a98d-422d-8f55-3a1c4a41d952-bound-sa-token\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.860576 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.860530 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09ac04c7-a98d-422d-8f55-3a1c4a41d952-ca-trust-extracted\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.860576 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.860573 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09ac04c7-a98d-422d-8f55-3a1c4a41d952-image-registry-private-configuration\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.860731 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.860603 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09ac04c7-a98d-422d-8f55-3a1c4a41d952-trusted-ca\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.860731 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.860624 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09ac04c7-a98d-422d-8f55-3a1c4a41d952-registry-certificates\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.860731 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.860645 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09ac04c7-a98d-422d-8f55-3a1c4a41d952-installation-pull-secrets\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.861920 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.861890 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09ac04c7-a98d-422d-8f55-3a1c4a41d952-registry-certificates\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.862112 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.862082 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09ac04c7-a98d-422d-8f55-3a1c4a41d952-trusted-ca\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.862222 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.862198 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09ac04c7-a98d-422d-8f55-3a1c4a41d952-ca-trust-extracted\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.863075 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.863052 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09ac04c7-a98d-422d-8f55-3a1c4a41d952-registry-tls\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.863513 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.863471 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09ac04c7-a98d-422d-8f55-3a1c4a41d952-image-registry-private-configuration\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.864298 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.864273 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09ac04c7-a98d-422d-8f55-3a1c4a41d952-installation-pull-secrets\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.883487 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.883457 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q6sn\" (UniqueName: \"kubernetes.io/projected/09ac04c7-a98d-422d-8f55-3a1c4a41d952-kube-api-access-8q6sn\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.883999 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.883979 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09ac04c7-a98d-422d-8f55-3a1c4a41d952-bound-sa-token\") pod \"image-registry-84f677497d-dp9bl\" (UID: \"09ac04c7-a98d-422d-8f55-3a1c4a41d952\") " pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:57.960589 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:57.960556 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:58.144567 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.144491 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-79n4w"] Oct 31 12:59:58.152159 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:58.152114 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd02e70d_668b_42d9_928e_ca8ddfd352d0.slice/crio-53febcad7d5c075e298e29315d0fa5609c8558e8c46424e9ed618b26dcfceb19 WatchSource:0}: Error finding container 53febcad7d5c075e298e29315d0fa5609c8558e8c46424e9ed618b26dcfceb19: Status 404 returned error can't find the container with id 53febcad7d5c075e298e29315d0fa5609c8558e8c46424e9ed618b26dcfceb19 Oct 31 12:59:58.258385 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.258357 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f677497d-dp9bl"] Oct 31 12:59:58.261353 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:58.261330 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ac04c7_a98d_422d_8f55_3a1c4a41d952.slice/crio-951fe318dafa7ca1234c8f9111b325fcbf85976b36c3b7193c5549ba6b4d2865 WatchSource:0}: Error finding container 951fe318dafa7ca1234c8f9111b325fcbf85976b36c3b7193c5549ba6b4d2865: Status 404 returned error can't find the container with id 951fe318dafa7ca1234c8f9111b325fcbf85976b36c3b7193c5549ba6b4d2865 Oct 31 12:59:58.352264 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.352230 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp"] Oct 31 12:59:58.355266 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.355243 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp" Oct 31 12:59:58.359619 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.359598 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qm68j\"" Oct 31 12:59:58.359619 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.359618 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 12:59:58.367643 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.367622 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp"] Oct 31 12:59:58.388555 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.388528 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f677497d-dp9bl" event={"ID":"09ac04c7-a98d-422d-8f55-3a1c4a41d952","Type":"ContainerStarted","Data":"a31eeb0b3a14e74fa0fe9df57238d2572a1354b60d0f765562b90b32f0cb9268"} Oct 31 12:59:58.388680 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.388567 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f677497d-dp9bl" event={"ID":"09ac04c7-a98d-422d-8f55-3a1c4a41d952","Type":"ContainerStarted","Data":"951fe318dafa7ca1234c8f9111b325fcbf85976b36c3b7193c5549ba6b4d2865"} Oct 31 12:59:58.388735 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.388712 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 12:59:58.389962 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.389940 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-79n4w" event={"ID":"cd02e70d-668b-42d9-928e-ca8ddfd352d0","Type":"ContainerStarted","Data":"53febcad7d5c075e298e29315d0fa5609c8558e8c46424e9ed618b26dcfceb19"} Oct 31 12:59:58.415905 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.415829 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84f677497d-dp9bl" podStartSLOduration=1.415817847 podStartE2EDuration="1.415817847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:58.415086065 +0000 UTC m=+15.260252498" watchObservedRunningTime="2026-10-31 12:59:58.415817847 +0000 UTC m=+15.260984259" Oct 31 12:59:58.464592 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.464563 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1d238636-8cb3-4923-b4b0-d58909ef45d2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lc8jp\" (UID: \"1d238636-8cb3-4923-b4b0-d58909ef45d2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp" Oct 31 12:59:58.565623 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.565585 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1d238636-8cb3-4923-b4b0-d58909ef45d2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lc8jp\" (UID: \"1d238636-8cb3-4923-b4b0-d58909ef45d2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp" Oct 31 12:59:58.568025 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.568003 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1d238636-8cb3-4923-b4b0-d58909ef45d2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lc8jp\" (UID: \"1d238636-8cb3-4923-b4b0-d58909ef45d2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp" Oct 31 12:59:58.666310 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.666233 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp" Oct 31 12:59:58.941157 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:58.941078 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp"] Oct 31 12:59:58.944266 ip-10-0-137-206 kubenswrapper[4542]: W1031 12:59:58.944234 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d238636_8cb3_4923_b4b0_d58909ef45d2.slice/crio-fa49da6771ce289ce37fe75395569919762714083827b6430457f4037fa413eb WatchSource:0}: Error finding container fa49da6771ce289ce37fe75395569919762714083827b6430457f4037fa413eb: Status 404 returned error can't find the container with id fa49da6771ce289ce37fe75395569919762714083827b6430457f4037fa413eb Oct 31 12:59:59.395415 ip-10-0-137-206 kubenswrapper[4542]: I1031 12:59:59.395373 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp" event={"ID":"1d238636-8cb3-4923-b4b0-d58909ef45d2","Type":"ContainerStarted","Data":"fa49da6771ce289ce37fe75395569919762714083827b6430457f4037fa413eb"} Oct 31 13:00:00.400920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:00.400886 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp" event={"ID":"1d238636-8cb3-4923-b4b0-d58909ef45d2","Type":"ContainerStarted","Data":"39fb7bebc78cf5b6d6e86c4f376720d9caf15eafaf0ea9f78bbb345cda609afa"} Oct 31 13:00:00.401401 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:00.401096 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp" Oct 31 13:00:00.406422 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:00.406398 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp" Oct 31 13:00:00.421345 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:00.421305 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lc8jp" podStartSLOduration=1.318873039 podStartE2EDuration="2.421293859s" totalImagesPullingTime="1.10242082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:00.420514596 +0000 UTC m=+17.265681014" watchObservedRunningTime="2026-10-31 13:00:00.421293859 +0000 UTC m=+17.266460268" Oct 31 13:00:03.050540 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:03.050507 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rwtn6" Oct 31 13:00:04.379294 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:04.379265 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ltf7k" Oct 31 13:00:06.919702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:06.919674 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k7d4p"] Oct 31 13:00:06.924520 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:06.924498 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:06.938136 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:06.938115 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 13:00:06.938136 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:06.938115 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 13:00:06.938281 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:06.938171 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l8r9q\"" Oct 31 13:00:07.004964 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.004942 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 13:00:07.005094 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.004994 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 13:00:07.005363 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.005338 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 13:00:07.005464 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.005426 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 13:00:07.025671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.025645 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6twj\" (UniqueName: \"kubernetes.io/projected/417e2a85-64e8-451c-b6d9-f553c70d6fe8-kube-api-access-t6twj\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.025789 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.025689 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.025789 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.025717 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-textfile\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.025789 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.025777 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/417e2a85-64e8-451c-b6d9-f553c70d6fe8-metrics-client-ca\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.025953 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.025846 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-tls\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.025953 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.025924 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.026042 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.025952 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-wtmp\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.026042 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.025976 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/417e2a85-64e8-451c-b6d9-f553c70d6fe8-root\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.026101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.026083 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/417e2a85-64e8-451c-b6d9-f553c70d6fe8-sys\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.126694 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.126582 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6twj\" (UniqueName: \"kubernetes.io/projected/417e2a85-64e8-451c-b6d9-f553c70d6fe8-kube-api-access-t6twj\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.126694 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.126633 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.126869 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.126761 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-textfile\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.126869 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.126834 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/417e2a85-64e8-451c-b6d9-f553c70d6fe8-metrics-client-ca\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127004 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.126907 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-tls\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127004 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.126964 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127004 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.126993 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-wtmp\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.127019 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/417e2a85-64e8-451c-b6d9-f553c70d6fe8-root\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.127046 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/417e2a85-64e8-451c-b6d9-f553c70d6fe8-sys\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.127125 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/417e2a85-64e8-451c-b6d9-f553c70d6fe8-sys\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127299 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.127175 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/417e2a85-64e8-451c-b6d9-f553c70d6fe8-root\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127299 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.127189 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-wtmp\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127402 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.127311 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-textfile\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127577 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.127559 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/417e2a85-64e8-451c-b6d9-f553c70d6fe8-metrics-client-ca\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.127668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.127576 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.129536 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.129512 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-tls\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.129815 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.129794 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/417e2a85-64e8-451c-b6d9-f553c70d6fe8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.140011 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.139989 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6twj\" (UniqueName: \"kubernetes.io/projected/417e2a85-64e8-451c-b6d9-f553c70d6fe8-kube-api-access-t6twj\") pod \"node-exporter-k7d4p\" (UID: \"417e2a85-64e8-451c-b6d9-f553c70d6fe8\") " pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.241747 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.241688 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k7d4p" Oct 31 13:00:07.252829 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:00:07.252803 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417e2a85_64e8_451c_b6d9_f553c70d6fe8.slice/crio-b9681010deb0d2561015345c7d3d2ca7dc19e6ecd226ac98e0838259de7d3911 WatchSource:0}: Error finding container b9681010deb0d2561015345c7d3d2ca7dc19e6ecd226ac98e0838259de7d3911: Status 404 returned error can't find the container with id b9681010deb0d2561015345c7d3d2ca7dc19e6ecd226ac98e0838259de7d3911 Oct 31 13:00:07.425866 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:07.425826 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7d4p" event={"ID":"417e2a85-64e8-451c-b6d9-f553c70d6fe8","Type":"ContainerStarted","Data":"b9681010deb0d2561015345c7d3d2ca7dc19e6ecd226ac98e0838259de7d3911"} Oct 31 13:00:09.434740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:09.434698 4542 generic.go:356] "Generic (PLEG): container finished" podID="417e2a85-64e8-451c-b6d9-f553c70d6fe8" containerID="6c58798ac3607791db0b4c3c07fa83501de2ab711713182e82d8c79ae4d618ad" exitCode=0 Oct 31 13:00:09.435153 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:09.434753 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7d4p" event={"ID":"417e2a85-64e8-451c-b6d9-f553c70d6fe8","Type":"ContainerDied","Data":"6c58798ac3607791db0b4c3c07fa83501de2ab711713182e82d8c79ae4d618ad"} Oct 31 13:00:10.440298 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:10.440264 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7d4p" event={"ID":"417e2a85-64e8-451c-b6d9-f553c70d6fe8","Type":"ContainerStarted","Data":"f5b452526405e148501cb4c87afcce3db73f04eda13efee8af9e935754ed96e3"} Oct 31 13:00:10.440802 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:10.440325 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7d4p" event={"ID":"417e2a85-64e8-451c-b6d9-f553c70d6fe8","Type":"ContainerStarted","Data":"39daf86ca2f896ddcedd347a607e9359a5e2cd22b7ea8a76ac8ba053f9bda533"} Oct 31 13:00:13.319936 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.316204 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k7d4p" podStartSLOduration=5.830150558 podStartE2EDuration="7.316157699s" totalImagesPullingTime="1.405714393s" totalInitContainerRuntime="80.292748ms" isStatefulPod=true podCreationTimestamp="2026-10-31 13:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:10.465450676 +0000 UTC m=+27.310617093" watchObservedRunningTime="2026-10-31 13:00:13.316157699 +0000 UTC m=+30.161324122" Oct 31 13:00:13.319936 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.317381 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:00:13.347620 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.347585 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:00:13.347790 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.347673 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.353813 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.353787 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 13:00:13.353947 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.353843 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 13:00:13.353947 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.353881 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 13:00:13.354587 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.354544 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 13:00:13.354587 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.354559 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 13:00:13.357338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.357317 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bf798\"" Oct 31 13:00:13.457215 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.457187 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 13:00:13.457438 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.457189 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 13:00:13.457438 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.457187 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 13:00:13.457657 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.457268 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 13:00:13.457657 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.457273 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 13:00:13.457861 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.457844 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 13:00:13.457993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.457976 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fh181l2udumbn\"" Oct 31 13:00:13.458057 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.458008 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 13:00:13.458057 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.458050 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 13:00:13.466444 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.466422 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 13:00:13.468051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.468032 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 13:00:13.476256 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476232 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476352 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476262 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476352 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476279 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvf6\" (UniqueName: \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-kube-api-access-qdvf6\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476352 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476296 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476400 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476451 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476538 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476574 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476605 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476660 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config-out\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476681 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476748 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476940 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476781 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476940 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476837 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.476940 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476870 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.477037 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.476977 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.477037 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.477009 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.477120 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.477072 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.477165 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.477129 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.477165 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.477158 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-web-config\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578350 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578269 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578350 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578316 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578350 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578353 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578382 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578410 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578437 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578457 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-web-config\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578505 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578536 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578558 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvf6\" (UniqueName: \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-kube-api-access-qdvf6\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578583 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.578640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578617 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.579061 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.578649 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.579257 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.579223 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.579382 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.579282 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.579382 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.579321 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.579382 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.579375 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config-out\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.579576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.579394 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.579576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.579405 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.579576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.579431 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.579576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.579456 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.579968 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.579944 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.580208 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.580183 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.581142 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.580455 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.581142 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.580802 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.582795 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.581626 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.582795 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.581939 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.582795 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.582232 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.582795 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.582493 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.583031 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.582925 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.583321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.583211 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.583321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.579321 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.583321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.583287 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.584394 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.584368 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config-out\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.584948 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.584921 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-web-config\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.585419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.585397 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.586336 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.586315 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.586663 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.586641 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.586993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.586975 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.599005 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.598986 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvf6\" (UniqueName: \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-kube-api-access-qdvf6\") pod \"prometheus-k8s-0\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:13.760576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:13.760469 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:14.081856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:14.081464 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:00:14.084032 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:00:14.084006 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0ba9c9_5d94_45f8_a5bb_5dfa7b4f0fcc.slice/crio-639814f21c84fccb7978c0b4769e824964154147bd6e914098bff8f0c0a67a35 WatchSource:0}: Error finding container 639814f21c84fccb7978c0b4769e824964154147bd6e914098bff8f0c0a67a35: Status 404 returned error can't find the container with id 639814f21c84fccb7978c0b4769e824964154147bd6e914098bff8f0c0a67a35 Oct 31 13:00:14.462115 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:14.462077 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerStarted","Data":"639814f21c84fccb7978c0b4769e824964154147bd6e914098bff8f0c0a67a35"} Oct 31 13:00:15.336455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:15.336422 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sjsfh" Oct 31 13:00:16.470667 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:16.470631 4542 generic.go:356] "Generic (PLEG): container finished" podID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerID="41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2" exitCode=0 Oct 31 13:00:16.471045 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:16.470723 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerDied","Data":"41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2"} Oct 31 13:00:19.402034 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:19.402000 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84f677497d-dp9bl" Oct 31 13:00:22.495337 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:22.495300 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerStarted","Data":"3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48"} Oct 31 13:00:22.495759 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:22.495378 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerStarted","Data":"4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236"} Oct 31 13:00:23.834189 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:00:23.834149 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:00:24.508263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:24.506596 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-79n4w" event={"ID":"cd02e70d-668b-42d9-928e-ca8ddfd352d0","Type":"ContainerStarted","Data":"e213b3532befa6d45e947c48cc0091d2a0a69d8243884d510a0510d81927003d"} Oct 31 13:00:24.508263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:24.506750 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-79n4w" Oct 31 13:00:24.516326 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:24.516235 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-79n4w" Oct 31 13:00:24.530180 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:24.530131 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-79n4w" podStartSLOduration=1.424963741 podStartE2EDuration="27.530112915s" totalImagesPullingTime="26.105149174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:24.528307756 +0000 UTC m=+41.373474175" watchObservedRunningTime="2026-10-31 13:00:24.530112915 +0000 UTC m=+41.375279334" Oct 31 13:00:25.520731 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:25.520630 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerStarted","Data":"13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76"} Oct 31 13:00:25.521150 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:25.520750 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerStarted","Data":"c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3"} Oct 31 13:00:25.521150 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:25.520794 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerStarted","Data":"b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952"} Oct 31 13:00:25.521150 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:25.520835 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerStarted","Data":"44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191"} Oct 31 13:00:25.574024 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:25.573965 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.682460251 podStartE2EDuration="12.573945066s" totalImagesPullingTime="9.878613709s" totalInitContainerRuntime="12.871106ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:25.571339405 +0000 UTC m=+42.416505823" watchObservedRunningTime="2026-10-31 13:00:25.573945066 +0000 UTC m=+42.419111486" Oct 31 13:00:28.760684 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:28.760597 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:41.331639 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:41.331609 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:41.372805 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:41.372782 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:41.591060 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:41.590991 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:49.197287 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.197254 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-q8jf6-vd797"] Oct 31 13:00:49.197884 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.197619 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-q8jf6-vd797" podUID="c6266aa7-b443-49e9-88a0-987228617831" containerName="insights-gathering" containerID="cri-o://d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384" gracePeriod=30 Oct 31 13:00:49.439769 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.439739 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 13:00:49.466629 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.466563 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-service-ca-bundle\") pod \"c6266aa7-b443-49e9-88a0-987228617831\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " Oct 31 13:00:49.466629 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.466597 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-trusted-ca-bundle\") pod \"c6266aa7-b443-49e9-88a0-987228617831\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " Oct 31 13:00:49.466629 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.466627 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6266aa7-b443-49e9-88a0-987228617831-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6266aa7-b443-49e9-88a0-987228617831-archives-path\") pod \"c6266aa7-b443-49e9-88a0-987228617831\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " Oct 31 13:00:49.466865 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.466733 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6266aa7-b443-49e9-88a0-987228617831-kube-api-access-z4fm9\" (UniqueName: \"kubernetes.io/projected/c6266aa7-b443-49e9-88a0-987228617831-kube-api-access-z4fm9\") pod \"c6266aa7-b443-49e9-88a0-987228617831\" (UID: \"c6266aa7-b443-49e9-88a0-987228617831\") " Oct 31 13:00:49.467009 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.466982 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6266aa7-b443-49e9-88a0-987228617831-archives-path" pod "c6266aa7-b443-49e9-88a0-987228617831" (UID: "c6266aa7-b443-49e9-88a0-987228617831"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:49.467009 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.466996 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-service-ca-bundle" pod "c6266aa7-b443-49e9-88a0-987228617831" (UID: "c6266aa7-b443-49e9-88a0-987228617831"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:49.467177 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.467124 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-trusted-ca-bundle" pod "c6266aa7-b443-49e9-88a0-987228617831" (UID: "c6266aa7-b443-49e9-88a0-987228617831"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:49.468920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.468898 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6266aa7-b443-49e9-88a0-987228617831-kube-api-access-z4fm9" pod "c6266aa7-b443-49e9-88a0-987228617831" (UID: "c6266aa7-b443-49e9-88a0-987228617831"). InnerVolumeSpecName "kube-api-access-z4fm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:49.567301 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.567278 4542 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-service-ca-bundle\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:49.567301 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.567300 4542 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6266aa7-b443-49e9-88a0-987228617831-trusted-ca-bundle\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:49.567459 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.567311 4542 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6266aa7-b443-49e9-88a0-987228617831-archives-path\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:49.567459 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.567319 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4fm9\" (UniqueName: \"kubernetes.io/projected/c6266aa7-b443-49e9-88a0-987228617831-kube-api-access-z4fm9\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:49.607048 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.607014 4542 generic.go:356] "Generic (PLEG): container finished" podID="c6266aa7-b443-49e9-88a0-987228617831" containerID="d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384" exitCode=2 Oct 31 13:00:49.607139 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.607059 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8jf6-vd797" Oct 31 13:00:49.607139 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.607062 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8jf6-vd797" event={"ID":"c6266aa7-b443-49e9-88a0-987228617831","Type":"ContainerDied","Data":"d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384"} Oct 31 13:00:49.607139 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.607110 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8jf6-vd797" event={"ID":"c6266aa7-b443-49e9-88a0-987228617831","Type":"ContainerDied","Data":"d79bd8837dd89a25bb7e97a9c72976dc77173457e3393580df8cfadd3e540867"} Oct 31 13:00:49.607139 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.607132 4542 scope.go:122] "RemoveContainer" containerID="d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384" Oct 31 13:00:49.615646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.615630 4542 scope.go:122] "RemoveContainer" containerID="d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384" Oct 31 13:00:49.615910 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:00:49.615888 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384\": container with ID starting with d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384 not found: ID does not exist" containerID="d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384" Oct 31 13:00:49.615974 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.615918 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384"} err="failed to get container status \"d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384\": rpc error: code = NotFound desc = could not find container \"d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384\": container with ID starting with d8daf14f5a7a991c8b1e1cd8c042fde58c1b8cd4d9e18368a204a26694599384 not found: ID does not exist" Oct 31 13:00:49.629324 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.629303 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-q8jf6-vd797"] Oct 31 13:00:49.633833 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:49.633810 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-q8jf6-vd797"] Oct 31 13:00:51.289270 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:51.289239 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6266aa7-b443-49e9-88a0-987228617831" path="/var/lib/kubelet/pods/c6266aa7-b443-49e9-88a0-987228617831/volumes" Oct 31 13:00:51.779823 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:51.779792 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:00:51.780093 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:51.780064 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="prometheus" containerID="cri-o://4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236" gracePeriod=600 Oct 31 13:00:51.780231 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:51.780096 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy" containerID="cri-o://c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3" gracePeriod=600 Oct 31 13:00:51.780231 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:51.780127 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76" gracePeriod=600 Oct 31 13:00:51.780231 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:51.780104 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="thanos-sidecar" containerID="cri-o://44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191" gracePeriod=600 Oct 31 13:00:51.780231 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:51.780151 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy-web" containerID="cri-o://b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952" gracePeriod=600 Oct 31 13:00:51.780231 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:51.780173 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="config-reloader" containerID="cri-o://3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48" gracePeriod=600 Oct 31 13:00:52.622202 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:52.622170 4542 generic.go:356] "Generic (PLEG): container finished" podID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerID="13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76" exitCode=0 Oct 31 13:00:52.622202 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:52.622198 4542 generic.go:356] "Generic (PLEG): container finished" podID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerID="c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3" exitCode=0 Oct 31 13:00:52.622202 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:52.622205 4542 generic.go:356] "Generic (PLEG): container finished" podID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerID="44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191" exitCode=0 Oct 31 13:00:52.622643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:52.622211 4542 generic.go:356] "Generic (PLEG): container finished" podID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerID="3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48" exitCode=0 Oct 31 13:00:52.622643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:52.622217 4542 generic.go:356] "Generic (PLEG): container finished" podID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerID="4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236" exitCode=0 Oct 31 13:00:52.622643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:52.622241 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerDied","Data":"13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76"} Oct 31 13:00:52.622643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:52.622301 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerDied","Data":"c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3"} Oct 31 13:00:52.622643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:52.622327 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerDied","Data":"44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191"} Oct 31 13:00:52.622643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:52.622352 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerDied","Data":"3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48"} Oct 31 13:00:52.622643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:52.622375 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerDied","Data":"4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236"} Oct 31 13:00:53.061235 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.061203 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.093610 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093585 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-thanos-prometheus-http-client-file\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.093712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093615 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.093712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093642 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-metrics-client-certs\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.093712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093673 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-2\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.093712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093700 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-kube-rbac-proxy\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.093937 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093730 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.093937 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093757 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-grpc-tls\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.093937 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093803 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-trusted-ca-bundle\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.093937 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093830 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-kubelet-serving-ca-bundle\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.093937 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093864 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-web-config\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-web-config\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.093937 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093909 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config-out\" (UniqueName: \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config-out\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.094240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.093948 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-0\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.094240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094006 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.094240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094057 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-metrics-client-ca\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.094240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094060 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-2" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:53.094240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094088 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-db\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.094240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094113 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-tls\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.094240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094146 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-1\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.094240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094184 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-serving-certs-ca-bundle\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.094240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094218 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-kube-api-access-qdvf6\" (UniqueName: \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-kube-api-access-qdvf6\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.094712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094247 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-tls-assets\" (UniqueName: \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-tls-assets\") pod \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\" (UID: \"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc\") " Oct 31 13:00:53.094712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094449 4542 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.094712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.094631 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-kubelet-serving-ca-bundle" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:53.095081 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.095029 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-trusted-ca-bundle" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:53.096404 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.096378 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:53.096596 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.096509 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-kube-rbac-proxy-web" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:53.096876 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.096793 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-tls-assets" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:53.097015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.096952 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-1" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:53.097284 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.097159 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-metrics-client-ca" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:53.097370 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.097339 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-serving-certs-ca-bundle" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:53.097770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.097745 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config-out" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:53.098091 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.098066 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-db" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:00:53.098172 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.098142 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-thanos-prometheus-http-client-file" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:53.098238 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.098195 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-metrics-client-certs" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:53.098238 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.098206 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-kube-rbac-proxy" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:53.098355 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.098256 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-grpc-tls" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:53.098620 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.098587 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-0" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:53.098821 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.098788 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-tls" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:53.099435 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.099412 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-kube-api-access-qdvf6" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "kube-api-access-qdvf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:53.100016 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.099995 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-thanos-sidecar-tls" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:53.109295 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.109268 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-web-config" pod "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" (UID: "db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:53.195683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195619 4542 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195644 4542 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-metrics-client-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195655 4542 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-db\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195663 4542 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195672 4542 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195680 4542 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195689 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvf6\" (UniqueName: \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-kube-api-access-qdvf6\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195698 4542 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-tls-assets\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195707 4542 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195716 4542 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195724 4542 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-metrics-client-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195733 4542 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-kube-rbac-proxy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195741 4542 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195749 4542 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-secret-grpc-tls\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195757 4542 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195765 4542 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195773 4542 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-web-config\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195782 4542 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-config-out\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.195931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.195792 4542 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:53.628447 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.628408 4542 generic.go:356] "Generic (PLEG): container finished" podID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerID="b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952" exitCode=0 Oct 31 13:00:53.628820 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.628471 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerDied","Data":"b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952"} Oct 31 13:00:53.628820 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.628523 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.628820 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.628597 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc","Type":"ContainerDied","Data":"639814f21c84fccb7978c0b4769e824964154147bd6e914098bff8f0c0a67a35"} Oct 31 13:00:53.628820 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.628647 4542 scope.go:122] "RemoveContainer" containerID="13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76" Oct 31 13:00:53.636140 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.636123 4542 scope.go:122] "RemoveContainer" containerID="c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3" Oct 31 13:00:53.642458 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.642444 4542 scope.go:122] "RemoveContainer" containerID="b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952" Oct 31 13:00:53.648622 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.648605 4542 scope.go:122] "RemoveContainer" containerID="44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191" Oct 31 13:00:53.653862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.653838 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:00:53.654943 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.654928 4542 scope.go:122] "RemoveContainer" containerID="3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48" Oct 31 13:00:53.660934 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.660914 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:00:53.661909 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.661891 4542 scope.go:122] "RemoveContainer" containerID="4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236" Oct 31 13:00:53.668348 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.668329 4542 scope.go:122] "RemoveContainer" containerID="41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2" Oct 31 13:00:53.674372 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.674357 4542 scope.go:122] "RemoveContainer" containerID="13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76" Oct 31 13:00:53.674709 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:00:53.674690 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76\": container with ID starting with 13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76 not found: ID does not exist" containerID="13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76" Oct 31 13:00:53.674797 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.674720 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76"} err="failed to get container status \"13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76\": rpc error: code = NotFound desc = could not find container \"13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76\": container with ID starting with 13d19f430f86fc783a60eb94e16fc35fd0cde645192a1b88e3b6547eb3204c76 not found: ID does not exist" Oct 31 13:00:53.674797 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.674744 4542 scope.go:122] "RemoveContainer" containerID="c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3" Oct 31 13:00:53.675011 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:00:53.674994 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3\": container with ID starting with c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3 not found: ID does not exist" containerID="c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3" Oct 31 13:00:53.675048 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.675017 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3"} err="failed to get container status \"c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3\": rpc error: code = NotFound desc = could not find container \"c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3\": container with ID starting with c16d52f6649814f636e274d1bebec2ae2f653e9795a1e07eb4b86293b74decb3 not found: ID does not exist" Oct 31 13:00:53.675048 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.675031 4542 scope.go:122] "RemoveContainer" containerID="b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952" Oct 31 13:00:53.675263 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:00:53.675243 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952\": container with ID starting with b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952 not found: ID does not exist" containerID="b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952" Oct 31 13:00:53.675307 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.675268 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952"} err="failed to get container status \"b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952\": rpc error: code = NotFound desc = could not find container \"b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952\": container with ID starting with b45b501a72019b5c00e6ef86c819c283d9f3b1f020100fa67ced230161c88952 not found: ID does not exist" Oct 31 13:00:53.675307 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.675289 4542 scope.go:122] "RemoveContainer" containerID="44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191" Oct 31 13:00:53.675510 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:00:53.675492 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191\": container with ID starting with 44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191 not found: ID does not exist" containerID="44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191" Oct 31 13:00:53.675548 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.675515 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191"} err="failed to get container status \"44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191\": rpc error: code = NotFound desc = could not find container \"44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191\": container with ID starting with 44fe810e8c0b47e43fa9baaa363376f93400c96c182febb4030bca94d3357191 not found: ID does not exist" Oct 31 13:00:53.675548 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.675528 4542 scope.go:122] "RemoveContainer" containerID="3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48" Oct 31 13:00:53.675772 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:00:53.675756 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48\": container with ID starting with 3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48 not found: ID does not exist" containerID="3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48" Oct 31 13:00:53.675813 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.675775 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48"} err="failed to get container status \"3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48\": rpc error: code = NotFound desc = could not find container \"3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48\": container with ID starting with 3c46833ed45986a00fb5f7ba3b5de7455134bfdf9106045db0fe104e9645da48 not found: ID does not exist" Oct 31 13:00:53.675813 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.675788 4542 scope.go:122] "RemoveContainer" containerID="4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236" Oct 31 13:00:53.676014 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:00:53.675999 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236\": container with ID starting with 4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236 not found: ID does not exist" containerID="4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236" Oct 31 13:00:53.676052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.676016 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236"} err="failed to get container status \"4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236\": rpc error: code = NotFound desc = could not find container \"4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236\": container with ID starting with 4d8062ae46193f6e80d8031620b2b99d16db471aa16e0227492565e751d06236 not found: ID does not exist" Oct 31 13:00:53.676052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.676028 4542 scope.go:122] "RemoveContainer" containerID="41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2" Oct 31 13:00:53.676226 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:00:53.676212 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2\": container with ID starting with 41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2 not found: ID does not exist" containerID="41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2" Oct 31 13:00:53.676259 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.676230 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2"} err="failed to get container status \"41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2\": rpc error: code = NotFound desc = could not find container \"41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2\": container with ID starting with 41586d83d5d073fa5a42fdbf4a457ab2ace6e61ddbfc6e649fbc1abe442179d2 not found: ID does not exist" Oct 31 13:00:53.695560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695542 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:00:53.695782 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695771 4542 cpu_manager.go:414] "containerMap: removing container" podUID="c6266aa7-b443-49e9-88a0-987228617831" containerName="insights-gathering" Oct 31 13:00:53.695822 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695784 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6266aa7-b443-49e9-88a0-987228617831" containerName="insights-gathering" Oct 31 13:00:53.695822 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695797 4542 cpu_manager.go:414] "containerMap: removing container" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="prometheus" Oct 31 13:00:53.695822 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695803 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="prometheus" Oct 31 13:00:53.695822 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695810 4542 cpu_manager.go:414] "containerMap: removing container" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="config-reloader" Oct 31 13:00:53.695822 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695814 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="config-reloader" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695822 4542 cpu_manager.go:414] "containerMap: removing container" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy-web" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695851 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy-web" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695860 4542 cpu_manager.go:414] "containerMap: removing container" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695865 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695877 4542 cpu_manager.go:414] "containerMap: removing container" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="thanos-sidecar" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695882 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="thanos-sidecar" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695887 4542 cpu_manager.go:414] "containerMap: removing container" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy-thanos" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695891 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy-thanos" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695900 4542 cpu_manager.go:414] "containerMap: removing container" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="init-config-reloader" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695904 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="init-config-reloader" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695943 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6266aa7-b443-49e9-88a0-987228617831" containerName="insights-gathering" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695953 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="prometheus" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695959 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="config-reloader" Oct 31 13:00:53.695962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695965 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy-web" Oct 31 13:00:53.696367 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695971 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy" Oct 31 13:00:53.696367 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695977 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="thanos-sidecar" Oct 31 13:00:53.696367 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.695982 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" containerName="kube-rbac-proxy-thanos" Oct 31 13:00:53.715720 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.715702 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.725855 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.725811 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 13:00:53.725855 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.725831 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 13:00:53.726051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.725864 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 13:00:53.726051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.725974 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 13:00:53.726051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.726027 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 13:00:53.726051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.726051 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 13:00:53.726685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.726667 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:00:53.726915 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.726897 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 13:00:53.727006 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.726930 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 13:00:53.727006 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.726956 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 13:00:53.727006 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.726962 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 13:00:53.727006 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.726980 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 13:00:53.727393 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.727375 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 13:00:53.727523 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.727426 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fh181l2udumbn\"" Oct 31 13:00:53.730694 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.730675 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bf798\"" Oct 31 13:00:53.733828 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.733811 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 13:00:53.733985 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.733967 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 13:00:53.734107 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.734091 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 13:00:53.800171 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800142 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800171 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800171 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800188 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800203 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800257 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800291 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800469 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800366 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800469 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800392 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800469 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800444 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-config-out\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800469 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800462 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800528 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800553 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800569 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstff\" (UniqueName: \"kubernetes.io/projected/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-kube-api-access-sstff\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800588 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800618 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800834 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800683 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-web-config\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800834 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800706 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800834 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800725 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-config\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800834 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800745 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.800834 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.800764 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.830254 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:00:53.830223 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:00:53.901794 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.901772 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.901895 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.901813 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.901895 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.901838 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.901895 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.901860 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.902058 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.901975 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.902058 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.902013 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.902178 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.902044 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.902333 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.902317 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.902471 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.902455 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-config-out\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.902728 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.902701 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.902862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.902727 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.902994 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.902976 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.903155 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.903117 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.903264 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.903250 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstff\" (UniqueName: \"kubernetes.io/projected/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-kube-api-access-sstff\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.903359 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.903338 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.903449 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.903433 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.903578 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.903562 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.903696 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.903682 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-web-config\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.903809 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.903795 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.903889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.903867 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.903950 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.903516 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.904045 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.904031 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-config\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.904164 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.904150 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.904271 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.904255 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.904959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.904514 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.904959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.904919 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.905135 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.905006 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.905499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.905394 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.906259 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.906205 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.906502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.906464 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.907072 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.907045 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.907388 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.907368 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-config\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.907502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.907460 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.907954 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.907927 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-config-out\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.908076 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.908054 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.908184 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.908163 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.908292 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.908271 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-web-config\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.909454 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.909435 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.909540 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.909518 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:53.917110 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:53.917090 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstff\" (UniqueName: \"kubernetes.io/projected/f4b3ffd0-32aa-4fb0-b3a4-191895c46215-kube-api-access-sstff\") pod \"prometheus-k8s-0\" (UID: \"f4b3ffd0-32aa-4fb0-b3a4-191895c46215\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:54.121055 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:54.121022 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:00:54.445545 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:54.445522 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:00:54.449265 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:00:54.447708 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b3ffd0_32aa_4fb0_b3a4_191895c46215.slice/crio-f89548dfcdf366ae9b83f6a56d0b932349861c38b39eebbdedbe4d37aeb3e228 WatchSource:0}: Error finding container f89548dfcdf366ae9b83f6a56d0b932349861c38b39eebbdedbe4d37aeb3e228: Status 404 returned error can't find the container with id f89548dfcdf366ae9b83f6a56d0b932349861c38b39eebbdedbe4d37aeb3e228 Oct 31 13:00:54.633583 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:54.633553 4542 generic.go:356] "Generic (PLEG): container finished" podID="f4b3ffd0-32aa-4fb0-b3a4-191895c46215" containerID="f32bbb3c2ac76c34497fb5adb09ce5189346cdb6532accfe5c7528c1e88fdd19" exitCode=0 Oct 31 13:00:54.633902 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:54.633593 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4b3ffd0-32aa-4fb0-b3a4-191895c46215","Type":"ContainerDied","Data":"f32bbb3c2ac76c34497fb5adb09ce5189346cdb6532accfe5c7528c1e88fdd19"} Oct 31 13:00:54.633902 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:54.633648 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4b3ffd0-32aa-4fb0-b3a4-191895c46215","Type":"ContainerStarted","Data":"f89548dfcdf366ae9b83f6a56d0b932349861c38b39eebbdedbe4d37aeb3e228"} Oct 31 13:00:55.289136 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:55.289048 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc" path="/var/lib/kubelet/pods/db0ba9c9-5d94-45f8-a5bb-5dfa7b4f0fcc/volumes" Oct 31 13:00:55.640195 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:55.640158 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4b3ffd0-32aa-4fb0-b3a4-191895c46215","Type":"ContainerStarted","Data":"d4bd637446f7180cabbf897742e2c4f4aeff2e4f8cd1ebc6073c383f71532d73"} Oct 31 13:00:55.640697 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:55.640221 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4b3ffd0-32aa-4fb0-b3a4-191895c46215","Type":"ContainerStarted","Data":"e5880d3eccf75221788cab228d8a37de8f480f192c53ae22fd315ef1f51ca98d"} Oct 31 13:00:55.640697 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:55.640245 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4b3ffd0-32aa-4fb0-b3a4-191895c46215","Type":"ContainerStarted","Data":"7f441c92fab862b261df399b7e6e3b04e062c25616f05708c80ba74d66e19dda"} Oct 31 13:00:55.640697 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:55.640268 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4b3ffd0-32aa-4fb0-b3a4-191895c46215","Type":"ContainerStarted","Data":"72e4f1311c0f85f09a4aa698842fb806b6fe884b7918cd9d9b9a33ce379cc397"} Oct 31 13:00:55.640697 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:55.640291 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4b3ffd0-32aa-4fb0-b3a4-191895c46215","Type":"ContainerStarted","Data":"aad89772aa67d6c5414473d05ea9d63b74b3d7a4d862649e467e98a302501726"} Oct 31 13:00:55.640697 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:55.640321 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f4b3ffd0-32aa-4fb0-b3a4-191895c46215","Type":"ContainerStarted","Data":"27dce181ac5f7567da749058ac453d1380ecf5f7a5388a5efd4f2c3b032ca30b"} Oct 31 13:00:55.678363 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:55.678310 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.664777239 podStartE2EDuration="2.678293048s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.515809ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:55.676605408 +0000 UTC m=+72.521771825" watchObservedRunningTime="2026-10-31 13:00:55.678293048 +0000 UTC m=+72.523459463" Oct 31 13:00:59.122174 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:00:59.122138 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:01:23.829828 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:01:23.829788 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:01:53.829586 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:01:53.829538 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:01:54.121849 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:01:54.121743 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:01:54.136847 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:01:54.136822 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:01:54.852923 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:01:54.852896 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:02:23.830569 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:02:23.830525 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:02:53.830438 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:02:53.830399 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:03:23.828801 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:03:23.828765 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:03:53.830237 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:03:53.830201 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:04:23.829897 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:04:23.829859 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:04:43.294554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:04:43.294520 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:04:43.295082 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:04:43.294800 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:04:53.830453 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:04:53.830416 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:05:10.974872 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:10.974795 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr"] Oct 31 13:05:10.977045 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:10.977030 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:10.998811 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:10.998784 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca46985-7193-421f-8eb3-163824750a95-apiservice-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:10.998920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:10.998861 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca46985-7193-421f-8eb3-163824750a95-metrics-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:10.998920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:10.998887 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zzr4\" (UniqueName: \"kubernetes.io/projected/4ca46985-7193-421f-8eb3-163824750a95-kube-api-access-6zzr4\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:10.998993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:10.998961 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca46985-7193-421f-8eb3-163824750a95-webhook-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:10.999030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:10.998998 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca46985-7193-421f-8eb3-163824750a95-manifests\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:10.999030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:10.999022 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca46985-7193-421f-8eb3-163824750a95-tmp\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.010772 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.010747 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr"] Oct 31 13:05:11.055182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.055161 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 13:05:11.055524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.055504 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 13:05:11.055602 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.055573 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9qk9g\"" Oct 31 13:05:11.055602 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.055588 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 13:05:11.055702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.055603 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 13:05:11.055702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.055576 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 13:05:11.100046 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.100022 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca46985-7193-421f-8eb3-163824750a95-metrics-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.100144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.100049 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zzr4\" (UniqueName: \"kubernetes.io/projected/4ca46985-7193-421f-8eb3-163824750a95-kube-api-access-6zzr4\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.100144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.100086 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca46985-7193-421f-8eb3-163824750a95-webhook-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.100144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.100121 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca46985-7193-421f-8eb3-163824750a95-manifests\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.100344 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.100143 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca46985-7193-421f-8eb3-163824750a95-tmp\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.100344 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.100169 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca46985-7193-421f-8eb3-163824750a95-apiservice-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.100658 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.100628 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca46985-7193-421f-8eb3-163824750a95-manifests\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.102627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.102598 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca46985-7193-421f-8eb3-163824750a95-tmp\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.102627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.102616 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca46985-7193-421f-8eb3-163824750a95-webhook-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.102807 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.102787 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca46985-7193-421f-8eb3-163824750a95-apiservice-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.102807 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.102801 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca46985-7193-421f-8eb3-163824750a95-metrics-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.116583 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.116562 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zzr4\" (UniqueName: \"kubernetes.io/projected/4ca46985-7193-421f-8eb3-163824750a95-kube-api-access-6zzr4\") pod \"opendatahub-operator-controller-manager-849556f89d-9n7pr\" (UID: \"4ca46985-7193-421f-8eb3-163824750a95\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.289509 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.289434 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:11.674013 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.673976 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr"] Oct 31 13:05:11.675001 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:05:11.674971 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca46985_7193_421f_8eb3_163824750a95.slice/crio-fcce9935653c6c8e8d5dd670fd65f8d89b8cf790646a1e5d392f506779ba7ea8 WatchSource:0}: Error finding container fcce9935653c6c8e8d5dd670fd65f8d89b8cf790646a1e5d392f506779ba7ea8: Status 404 returned error can't find the container with id fcce9935653c6c8e8d5dd670fd65f8d89b8cf790646a1e5d392f506779ba7ea8 Oct 31 13:05:11.676530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:11.676514 4542 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:05:12.469684 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:12.469653 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" event={"ID":"4ca46985-7193-421f-8eb3-163824750a95","Type":"ContainerStarted","Data":"fcce9935653c6c8e8d5dd670fd65f8d89b8cf790646a1e5d392f506779ba7ea8"} Oct 31 13:05:15.485553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:15.485454 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" event={"ID":"4ca46985-7193-421f-8eb3-163824750a95","Type":"ContainerStarted","Data":"50b24ebf6150de98fe20f7c1f28e8391d75555b6d47bc0c05812066fd408acf6"} Oct 31 13:05:16.490096 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:16.490061 4542 generic.go:356] "Generic (PLEG): container finished" podID="4ca46985-7193-421f-8eb3-163824750a95" containerID="50b24ebf6150de98fe20f7c1f28e8391d75555b6d47bc0c05812066fd408acf6" exitCode=0 Oct 31 13:05:16.490495 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:16.490115 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" event={"ID":"4ca46985-7193-421f-8eb3-163824750a95","Type":"ContainerDied","Data":"50b24ebf6150de98fe20f7c1f28e8391d75555b6d47bc0c05812066fd408acf6"} Oct 31 13:05:17.495002 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:17.494965 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" event={"ID":"4ca46985-7193-421f-8eb3-163824750a95","Type":"ContainerStarted","Data":"0d86832cd5a919a2bb5e10909f8794287b507a2f91dc0255171e59d05a79401e"} Oct 31 13:05:17.495471 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:17.495075 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:17.542775 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:17.542725 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" podStartSLOduration=3.146811286 podStartE2EDuration="7.542710046s" totalImagesPullingTime="3.914935415s" totalInitContainerRuntime="480.963345ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:17.540437045 +0000 UTC m=+334.385603464" watchObservedRunningTime="2026-10-31 13:05:17.542710046 +0000 UTC m=+334.387876463" Oct 31 13:05:23.829831 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:05:23.829788 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:05:28.501757 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:28.501726 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-9n7pr" Oct 31 13:05:28.883107 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:28.883068 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj"] Oct 31 13:05:28.886312 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:28.886292 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" Oct 31 13:05:28.893113 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:28.893092 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-77xsp\"" Oct 31 13:05:28.893230 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:28.893115 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 13:05:28.893230 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:28.893188 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 13:05:28.900063 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:28.900038 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj"] Oct 31 13:05:28.916540 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:28.916516 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56256a5c-3090-4dd8-b0aa-ae268a9f9d56-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5wppj\" (UID: \"56256a5c-3090-4dd8-b0aa-ae268a9f9d56\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" Oct 31 13:05:28.916654 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:28.916565 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcsrq\" (UniqueName: \"kubernetes.io/projected/56256a5c-3090-4dd8-b0aa-ae268a9f9d56-kube-api-access-mcsrq\") pod \"cert-manager-cainjector-5c57bbd4fc-5wppj\" (UID: \"56256a5c-3090-4dd8-b0aa-ae268a9f9d56\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" Oct 31 13:05:29.017213 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:29.017175 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56256a5c-3090-4dd8-b0aa-ae268a9f9d56-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5wppj\" (UID: \"56256a5c-3090-4dd8-b0aa-ae268a9f9d56\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" Oct 31 13:05:29.017370 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:29.017229 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcsrq\" (UniqueName: \"kubernetes.io/projected/56256a5c-3090-4dd8-b0aa-ae268a9f9d56-kube-api-access-mcsrq\") pod \"cert-manager-cainjector-5c57bbd4fc-5wppj\" (UID: \"56256a5c-3090-4dd8-b0aa-ae268a9f9d56\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" Oct 31 13:05:29.030718 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:29.030688 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56256a5c-3090-4dd8-b0aa-ae268a9f9d56-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5wppj\" (UID: \"56256a5c-3090-4dd8-b0aa-ae268a9f9d56\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" Oct 31 13:05:29.036232 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:29.036208 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcsrq\" (UniqueName: \"kubernetes.io/projected/56256a5c-3090-4dd8-b0aa-ae268a9f9d56-kube-api-access-mcsrq\") pod \"cert-manager-cainjector-5c57bbd4fc-5wppj\" (UID: \"56256a5c-3090-4dd8-b0aa-ae268a9f9d56\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" Oct 31 13:05:29.210443 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:29.210364 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" Oct 31 13:05:29.486842 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:29.486762 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj"] Oct 31 13:05:29.490289 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:05:29.490262 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56256a5c_3090_4dd8_b0aa_ae268a9f9d56.slice/crio-461c0a87f1f89f574273c6513f4f05f333f8f2de4e905817b66e22069629ecb0 WatchSource:0}: Error finding container 461c0a87f1f89f574273c6513f4f05f333f8f2de4e905817b66e22069629ecb0: Status 404 returned error can't find the container with id 461c0a87f1f89f574273c6513f4f05f333f8f2de4e905817b66e22069629ecb0 Oct 31 13:05:29.538551 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:29.538520 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" event={"ID":"56256a5c-3090-4dd8-b0aa-ae268a9f9d56","Type":"ContainerStarted","Data":"461c0a87f1f89f574273c6513f4f05f333f8f2de4e905817b66e22069629ecb0"} Oct 31 13:05:33.294224 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.294191 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc"] Oct 31 13:05:33.297778 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.297755 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" Oct 31 13:05:33.307750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.307728 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-r55js\"" Oct 31 13:05:33.312402 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.312365 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc"] Oct 31 13:05:33.354752 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.354716 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brdjd\" (UniqueName: \"kubernetes.io/projected/65e8dcbf-3370-4090-a923-cd11a92a5749-kube-api-access-brdjd\") pod \"cert-manager-webhook-5dbbbf75cc-qqtbc\" (UID: \"65e8dcbf-3370-4090-a923-cd11a92a5749\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" Oct 31 13:05:33.354882 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.354776 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e8dcbf-3370-4090-a923-cd11a92a5749-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qqtbc\" (UID: \"65e8dcbf-3370-4090-a923-cd11a92a5749\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" Oct 31 13:05:33.455899 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.455877 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brdjd\" (UniqueName: \"kubernetes.io/projected/65e8dcbf-3370-4090-a923-cd11a92a5749-kube-api-access-brdjd\") pod \"cert-manager-webhook-5dbbbf75cc-qqtbc\" (UID: \"65e8dcbf-3370-4090-a923-cd11a92a5749\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" Oct 31 13:05:33.455994 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.455913 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e8dcbf-3370-4090-a923-cd11a92a5749-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qqtbc\" (UID: \"65e8dcbf-3370-4090-a923-cd11a92a5749\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" Oct 31 13:05:33.470557 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.470531 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e8dcbf-3370-4090-a923-cd11a92a5749-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qqtbc\" (UID: \"65e8dcbf-3370-4090-a923-cd11a92a5749\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" Oct 31 13:05:33.470757 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.470731 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brdjd\" (UniqueName: \"kubernetes.io/projected/65e8dcbf-3370-4090-a923-cd11a92a5749-kube-api-access-brdjd\") pod \"cert-manager-webhook-5dbbbf75cc-qqtbc\" (UID: \"65e8dcbf-3370-4090-a923-cd11a92a5749\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" Oct 31 13:05:33.554314 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.554242 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" event={"ID":"56256a5c-3090-4dd8-b0aa-ae268a9f9d56","Type":"ContainerStarted","Data":"acc3b3f209262e8d71b13ea38faf2eacf0fd0a49d20fc0ecaee2938c29691b90"} Oct 31 13:05:33.578104 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.578051 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5wppj" podStartSLOduration=1.671324029 podStartE2EDuration="5.578035814s" totalImagesPullingTime="3.906711785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:33.577415792 +0000 UTC m=+350.422582202" watchObservedRunningTime="2026-10-31 13:05:33.578035814 +0000 UTC m=+350.423202231" Oct 31 13:05:33.610946 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.610913 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" Oct 31 13:05:33.893106 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:33.892942 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc"] Oct 31 13:05:33.895601 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:05:33.895573 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65e8dcbf_3370_4090_a923_cd11a92a5749.slice/crio-e7cdfc7278da760381fbbe1a3eb31d7cda744fa5511f58a82f01805c093eea1d WatchSource:0}: Error finding container e7cdfc7278da760381fbbe1a3eb31d7cda744fa5511f58a82f01805c093eea1d: Status 404 returned error can't find the container with id e7cdfc7278da760381fbbe1a3eb31d7cda744fa5511f58a82f01805c093eea1d Oct 31 13:05:34.559431 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:34.559390 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" event={"ID":"65e8dcbf-3370-4090-a923-cd11a92a5749","Type":"ContainerStarted","Data":"6fd0f3183d25369fd2faaa74a68ef55ef6b95937441c37ec10507d0387fe46f1"} Oct 31 13:05:34.559431 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:34.559435 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" event={"ID":"65e8dcbf-3370-4090-a923-cd11a92a5749","Type":"ContainerStarted","Data":"e7cdfc7278da760381fbbe1a3eb31d7cda744fa5511f58a82f01805c093eea1d"} Oct 31 13:05:34.559957 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:34.559453 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" Oct 31 13:05:36.876839 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:36.876789 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" podStartSLOduration=3.87677371 podStartE2EDuration="3.87677371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:34.584614829 +0000 UTC m=+351.429781247" watchObservedRunningTime="2026-10-31 13:05:36.87677371 +0000 UTC m=+353.721940173" Oct 31 13:05:36.877276 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:36.877006 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-9qkwz"] Oct 31 13:05:36.880306 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:36.880284 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-9qkwz" Oct 31 13:05:36.889097 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:36.889075 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zxj88\"" Oct 31 13:05:36.894372 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:36.894348 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-9qkwz"] Oct 31 13:05:36.980821 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:36.980790 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944n8\" (UniqueName: \"kubernetes.io/projected/f3554073-ca31-4bd1-86d8-7af1b30cc75a-kube-api-access-944n8\") pod \"cert-manager-597b95cb84-9qkwz\" (UID: \"f3554073-ca31-4bd1-86d8-7af1b30cc75a\") " pod="cert-manager/cert-manager-597b95cb84-9qkwz" Oct 31 13:05:36.980954 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:36.980853 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3554073-ca31-4bd1-86d8-7af1b30cc75a-bound-sa-token\") pod \"cert-manager-597b95cb84-9qkwz\" (UID: \"f3554073-ca31-4bd1-86d8-7af1b30cc75a\") " pod="cert-manager/cert-manager-597b95cb84-9qkwz" Oct 31 13:05:37.082244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.082209 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3554073-ca31-4bd1-86d8-7af1b30cc75a-bound-sa-token\") pod \"cert-manager-597b95cb84-9qkwz\" (UID: \"f3554073-ca31-4bd1-86d8-7af1b30cc75a\") " pod="cert-manager/cert-manager-597b95cb84-9qkwz" Oct 31 13:05:37.082370 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.082261 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944n8\" (UniqueName: \"kubernetes.io/projected/f3554073-ca31-4bd1-86d8-7af1b30cc75a-kube-api-access-944n8\") pod \"cert-manager-597b95cb84-9qkwz\" (UID: \"f3554073-ca31-4bd1-86d8-7af1b30cc75a\") " pod="cert-manager/cert-manager-597b95cb84-9qkwz" Oct 31 13:05:37.100565 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.100537 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-944n8\" (UniqueName: \"kubernetes.io/projected/f3554073-ca31-4bd1-86d8-7af1b30cc75a-kube-api-access-944n8\") pod \"cert-manager-597b95cb84-9qkwz\" (UID: \"f3554073-ca31-4bd1-86d8-7af1b30cc75a\") " pod="cert-manager/cert-manager-597b95cb84-9qkwz" Oct 31 13:05:37.101637 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.101619 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3554073-ca31-4bd1-86d8-7af1b30cc75a-bound-sa-token\") pod \"cert-manager-597b95cb84-9qkwz\" (UID: \"f3554073-ca31-4bd1-86d8-7af1b30cc75a\") " pod="cert-manager/cert-manager-597b95cb84-9qkwz" Oct 31 13:05:37.191818 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.191756 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-9qkwz" Oct 31 13:05:37.359400 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.359363 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-776d875d45-6shpq"] Oct 31 13:05:37.362688 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.362672 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.374358 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.374330 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 13:05:37.375648 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.375608 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 13:05:37.375845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.375819 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 13:05:37.375845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.375819 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 13:05:37.375971 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.375924 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9nhp8\"" Oct 31 13:05:37.382427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.382408 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-776d875d45-6shpq"] Oct 31 13:05:37.385189 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.385159 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdtw\" (UniqueName: \"kubernetes.io/projected/f3080f7f-0fbf-40eb-9f27-89928be76699-kube-api-access-qfdtw\") pod \"kube-auth-proxy-776d875d45-6shpq\" (UID: \"f3080f7f-0fbf-40eb-9f27-89928be76699\") " pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.385286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.385217 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3080f7f-0fbf-40eb-9f27-89928be76699-tmp\") pod \"kube-auth-proxy-776d875d45-6shpq\" (UID: \"f3080f7f-0fbf-40eb-9f27-89928be76699\") " pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.385394 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.385371 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3080f7f-0fbf-40eb-9f27-89928be76699-tls-certs\") pod \"kube-auth-proxy-776d875d45-6shpq\" (UID: \"f3080f7f-0fbf-40eb-9f27-89928be76699\") " pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.486230 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.486155 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3080f7f-0fbf-40eb-9f27-89928be76699-tmp\") pod \"kube-auth-proxy-776d875d45-6shpq\" (UID: \"f3080f7f-0fbf-40eb-9f27-89928be76699\") " pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.486230 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.486216 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3080f7f-0fbf-40eb-9f27-89928be76699-tls-certs\") pod \"kube-auth-proxy-776d875d45-6shpq\" (UID: \"f3080f7f-0fbf-40eb-9f27-89928be76699\") " pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.486460 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.486248 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdtw\" (UniqueName: \"kubernetes.io/projected/f3080f7f-0fbf-40eb-9f27-89928be76699-kube-api-access-qfdtw\") pod \"kube-auth-proxy-776d875d45-6shpq\" (UID: \"f3080f7f-0fbf-40eb-9f27-89928be76699\") " pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.489184 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.489158 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3080f7f-0fbf-40eb-9f27-89928be76699-tmp\") pod \"kube-auth-proxy-776d875d45-6shpq\" (UID: \"f3080f7f-0fbf-40eb-9f27-89928be76699\") " pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.489364 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.489346 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3080f7f-0fbf-40eb-9f27-89928be76699-tls-certs\") pod \"kube-auth-proxy-776d875d45-6shpq\" (UID: \"f3080f7f-0fbf-40eb-9f27-89928be76699\") " pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.499084 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.499062 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-9qkwz"] Oct 31 13:05:37.501451 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:05:37.501418 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3554073_ca31_4bd1_86d8_7af1b30cc75a.slice/crio-9b032ac87c358e2e9c9ee0ef583a43c3b45c020e392b41eba2056d4a641c89e1 WatchSource:0}: Error finding container 9b032ac87c358e2e9c9ee0ef583a43c3b45c020e392b41eba2056d4a641c89e1: Status 404 returned error can't find the container with id 9b032ac87c358e2e9c9ee0ef583a43c3b45c020e392b41eba2056d4a641c89e1 Oct 31 13:05:37.504867 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.504850 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdtw\" (UniqueName: \"kubernetes.io/projected/f3080f7f-0fbf-40eb-9f27-89928be76699-kube-api-access-qfdtw\") pod \"kube-auth-proxy-776d875d45-6shpq\" (UID: \"f3080f7f-0fbf-40eb-9f27-89928be76699\") " pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.571529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.571501 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-9qkwz" event={"ID":"f3554073-ca31-4bd1-86d8-7af1b30cc75a","Type":"ContainerStarted","Data":"5c665329d6fed16de6f8bac413d451d5bb042f35ea9d0be1e3e51cc7ff5b02e7"} Oct 31 13:05:37.571662 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.571558 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-9qkwz" event={"ID":"f3554073-ca31-4bd1-86d8-7af1b30cc75a","Type":"ContainerStarted","Data":"9b032ac87c358e2e9c9ee0ef583a43c3b45c020e392b41eba2056d4a641c89e1"} Oct 31 13:05:37.650578 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.650519 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-9qkwz" podStartSLOduration=1.650504473 podStartE2EDuration="1.650504473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:37.647785547 +0000 UTC m=+354.492951965" watchObservedRunningTime="2026-10-31 13:05:37.650504473 +0000 UTC m=+354.495670888" Oct 31 13:05:37.674660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.674627 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" Oct 31 13:05:37.959516 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:37.959460 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-776d875d45-6shpq"] Oct 31 13:05:37.963549 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:05:37.963517 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3080f7f_0fbf_40eb_9f27_89928be76699.slice/crio-c572550dc3e8ac8d344982d924c055b9aab657604c7685542f3341b901b19557 WatchSource:0}: Error finding container c572550dc3e8ac8d344982d924c055b9aab657604c7685542f3341b901b19557: Status 404 returned error can't find the container with id c572550dc3e8ac8d344982d924c055b9aab657604c7685542f3341b901b19557 Oct 31 13:05:38.576151 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:38.576109 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" event={"ID":"f3080f7f-0fbf-40eb-9f27-89928be76699","Type":"ContainerStarted","Data":"c572550dc3e8ac8d344982d924c055b9aab657604c7685542f3341b901b19557"} Oct 31 13:05:40.564655 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:40.564628 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" Oct 31 13:05:40.584864 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:40.584840 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" event={"ID":"f3080f7f-0fbf-40eb-9f27-89928be76699","Type":"ContainerStarted","Data":"61c9ad8c2c67fef0c4bdddf599c37b1b26545767c714524641781d55155b92dd"} Oct 31 13:05:40.609067 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:40.608979 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-776d875d45-6shpq" podStartSLOduration=1.235070872 podStartE2EDuration="3.608962583s" totalImagesPullingTime="2.373891711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:40.608511423 +0000 UTC m=+357.453677836" watchObservedRunningTime="2026-10-31 13:05:40.608962583 +0000 UTC m=+357.454129004" Oct 31 13:05:41.694698 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.694667 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl"] Oct 31 13:05:41.697903 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.697887 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.714351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.714326 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl"] Oct 31 13:05:41.721623 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.721596 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dd88af6-332b-4505-8816-ac41c1421fb9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.721767 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.721651 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldm9l\" (UniqueName: \"kubernetes.io/projected/9dd88af6-332b-4505-8816-ac41c1421fb9-kube-api-access-ldm9l\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.721835 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.721782 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dd88af6-332b-4505-8816-ac41c1421fb9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.721898 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.721876 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9dd88af6-332b-4505-8816-ac41c1421fb9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.768716 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.768663 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 13:05:41.769014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.768892 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 13:05:41.769014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.768920 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-cdxqd\"" Oct 31 13:05:41.772554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.772538 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 13:05:41.822419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.822379 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dd88af6-332b-4505-8816-ac41c1421fb9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.822642 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.822437 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldm9l\" (UniqueName: \"kubernetes.io/projected/9dd88af6-332b-4505-8816-ac41c1421fb9-kube-api-access-ldm9l\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.822642 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.822471 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dd88af6-332b-4505-8816-ac41c1421fb9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.822642 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.822540 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9dd88af6-332b-4505-8816-ac41c1421fb9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.822995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.822972 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9dd88af6-332b-4505-8816-ac41c1421fb9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.823074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.823010 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dd88af6-332b-4505-8816-ac41c1421fb9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.823410 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.823387 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dd88af6-332b-4505-8816-ac41c1421fb9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:41.840408 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:41.840382 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldm9l\" (UniqueName: \"kubernetes.io/projected/9dd88af6-332b-4505-8816-ac41c1421fb9-kube-api-access-ldm9l\") pod \"kserve-module-controller-manager-7566cf5fc8-88dcl\" (UID: \"9dd88af6-332b-4505-8816-ac41c1421fb9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:42.009620 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:42.009521 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:42.292727 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:42.292654 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl"] Oct 31 13:05:42.295888 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:05:42.295858 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd88af6_332b_4505_8816_ac41c1421fb9.slice/crio-ab7424bd40fcc36855d94fda188be68081ae595ed0957cc09d482e32f246c917 WatchSource:0}: Error finding container ab7424bd40fcc36855d94fda188be68081ae595ed0957cc09d482e32f246c917: Status 404 returned error can't find the container with id ab7424bd40fcc36855d94fda188be68081ae595ed0957cc09d482e32f246c917 Oct 31 13:05:42.592954 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:42.592922 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" event={"ID":"9dd88af6-332b-4505-8816-ac41c1421fb9","Type":"ContainerStarted","Data":"ab7424bd40fcc36855d94fda188be68081ae595ed0957cc09d482e32f246c917"} Oct 31 13:05:43.392768 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.392738 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc"] Oct 31 13:05:43.397505 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.397485 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" Oct 31 13:05:43.405934 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.405905 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 13:05:43.406029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.405943 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 13:05:43.406029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.405949 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-22zfr\"" Oct 31 13:05:43.419804 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.419773 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc"] Oct 31 13:05:43.434400 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.434376 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6tr\" (UniqueName: \"kubernetes.io/projected/31c91a14-a352-471c-9857-d483ac2c758a-kube-api-access-vv6tr\") pod \"limitador-operator-controller-manager-5778bb4449-c6vhc\" (UID: \"31c91a14-a352-471c-9857-d483ac2c758a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" Oct 31 13:05:43.535792 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.535761 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6tr\" (UniqueName: \"kubernetes.io/projected/31c91a14-a352-471c-9857-d483ac2c758a-kube-api-access-vv6tr\") pod \"limitador-operator-controller-manager-5778bb4449-c6vhc\" (UID: \"31c91a14-a352-471c-9857-d483ac2c758a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" Oct 31 13:05:43.552151 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.552129 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6tr\" (UniqueName: \"kubernetes.io/projected/31c91a14-a352-471c-9857-d483ac2c758a-kube-api-access-vv6tr\") pod \"limitador-operator-controller-manager-5778bb4449-c6vhc\" (UID: \"31c91a14-a352-471c-9857-d483ac2c758a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" Oct 31 13:05:43.709996 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.709924 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" Oct 31 13:05:43.996553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:43.996457 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc"] Oct 31 13:05:44.000577 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:05:44.000550 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c91a14_a352_471c_9857_d483ac2c758a.slice/crio-5628d2da2a03f9310d24d920d7d6a9e8a72cc386bfc92c4b4acec937fd6cdea7 WatchSource:0}: Error finding container 5628d2da2a03f9310d24d920d7d6a9e8a72cc386bfc92c4b4acec937fd6cdea7: Status 404 returned error can't find the container with id 5628d2da2a03f9310d24d920d7d6a9e8a72cc386bfc92c4b4acec937fd6cdea7 Oct 31 13:05:44.602662 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:44.602630 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" event={"ID":"31c91a14-a352-471c-9857-d483ac2c758a","Type":"ContainerStarted","Data":"5628d2da2a03f9310d24d920d7d6a9e8a72cc386bfc92c4b4acec937fd6cdea7"} Oct 31 13:05:44.603946 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:44.603923 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" event={"ID":"9dd88af6-332b-4505-8816-ac41c1421fb9","Type":"ContainerStarted","Data":"1cc63e0fb62116c2de67a2d3fef5b40073636235be5092feaadf267e0b0c6a17"} Oct 31 13:05:44.604067 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:44.604056 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:05:44.640016 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:44.639974 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" podStartSLOduration=1.822574291 podStartE2EDuration="3.639961334s" totalImagesPullingTime="1.817387043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:44.63854496 +0000 UTC m=+361.483711380" watchObservedRunningTime="2026-10-31 13:05:44.639961334 +0000 UTC m=+361.485127751" Oct 31 13:05:48.623064 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:48.623023 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" event={"ID":"31c91a14-a352-471c-9857-d483ac2c758a","Type":"ContainerStarted","Data":"c20830439644e57c831c56cc01819950a57c52ff59fed0dc8ee3c37925352751"} Oct 31 13:05:48.623462 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:48.623143 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" Oct 31 13:05:48.645886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:48.645843 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" podStartSLOduration=1.830748797 podStartE2EDuration="5.645831445s" totalImagesPullingTime="3.815082648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:48.644561766 +0000 UTC m=+365.489728184" watchObservedRunningTime="2026-10-31 13:05:48.645831445 +0000 UTC m=+365.490997863" Oct 31 13:05:53.830535 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:05:53.830497 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:05:59.629210 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:05:59.629182 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" Oct 31 13:06:05.609625 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:05.609593 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-88dcl" Oct 31 13:06:23.830581 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:06:23.830544 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:06:26.163283 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.163248 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:06:26.171348 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.171329 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" Oct 31 13:06:26.178510 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.178463 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:06:26.184947 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.184929 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 13:06:26.187903 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.187878 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9kb5b\"" Oct 31 13:06:26.225551 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.225524 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:06:26.288343 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.288317 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkhj\" (UniqueName: \"kubernetes.io/projected/3c604ceb-6d74-416f-924d-22a8f5e835d7-kube-api-access-zlkhj\") pod \"limitador-limitador-7c9b87d574-hptgn\" (UID: \"3c604ceb-6d74-416f-924d-22a8f5e835d7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" Oct 31 13:06:26.288457 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.288366 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c604ceb-6d74-416f-924d-22a8f5e835d7-config-file\") pod \"limitador-limitador-7c9b87d574-hptgn\" (UID: \"3c604ceb-6d74-416f-924d-22a8f5e835d7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" Oct 31 13:06:26.389621 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.389591 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkhj\" (UniqueName: \"kubernetes.io/projected/3c604ceb-6d74-416f-924d-22a8f5e835d7-kube-api-access-zlkhj\") pod \"limitador-limitador-7c9b87d574-hptgn\" (UID: \"3c604ceb-6d74-416f-924d-22a8f5e835d7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" Oct 31 13:06:26.389763 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.389657 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c604ceb-6d74-416f-924d-22a8f5e835d7-config-file\") pod \"limitador-limitador-7c9b87d574-hptgn\" (UID: \"3c604ceb-6d74-416f-924d-22a8f5e835d7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" Oct 31 13:06:26.390266 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.390247 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c604ceb-6d74-416f-924d-22a8f5e835d7-config-file\") pod \"limitador-limitador-7c9b87d574-hptgn\" (UID: \"3c604ceb-6d74-416f-924d-22a8f5e835d7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" Oct 31 13:06:26.403656 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.403629 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkhj\" (UniqueName: \"kubernetes.io/projected/3c604ceb-6d74-416f-924d-22a8f5e835d7-kube-api-access-zlkhj\") pod \"limitador-limitador-7c9b87d574-hptgn\" (UID: \"3c604ceb-6d74-416f-924d-22a8f5e835d7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" Oct 31 13:06:26.483585 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.483465 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" Oct 31 13:06:26.774577 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:26.774553 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:06:26.776898 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:06:26.776873 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c604ceb_6d74_416f_924d_22a8f5e835d7.slice/crio-3e1ff2eec0857398ca03058876e701a195817860a3ee8b39f312155bf69741a9 WatchSource:0}: Error finding container 3e1ff2eec0857398ca03058876e701a195817860a3ee8b39f312155bf69741a9: Status 404 returned error can't find the container with id 3e1ff2eec0857398ca03058876e701a195817860a3ee8b39f312155bf69741a9 Oct 31 13:06:27.773544 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:27.773504 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" event={"ID":"3c604ceb-6d74-416f-924d-22a8f5e835d7","Type":"ContainerStarted","Data":"3e1ff2eec0857398ca03058876e701a195817860a3ee8b39f312155bf69741a9"} Oct 31 13:06:28.779366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:28.779329 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" event={"ID":"3c604ceb-6d74-416f-924d-22a8f5e835d7","Type":"ContainerStarted","Data":"8342854c16080682665b63d4ca24151e07a8afa902bfdd15715068878a96fd2d"} Oct 31 13:06:28.779783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:28.779415 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" Oct 31 13:06:28.802790 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:28.802743 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" podStartSLOduration=1.7750714969999999 podStartE2EDuration="2.802729555s" totalImagesPullingTime="1.027658058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:28.801322633 +0000 UTC m=+405.646489062" watchObservedRunningTime="2026-10-31 13:06:28.802729555 +0000 UTC m=+405.647895969" Oct 31 13:06:35.161784 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.161751 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7n9z4"] Oct 31 13:06:35.168790 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.168766 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" Oct 31 13:06:35.175516 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.175488 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7n9z4"] Oct 31 13:06:35.233719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.233683 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 13:06:35.233868 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.233749 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-84fdq\"" Oct 31 13:06:35.255170 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.255140 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk4jw\" (UniqueName: \"kubernetes.io/projected/b7ac2673-e6fe-4c31-b278-64345475da61-kube-api-access-hk4jw\") pod \"model-serving-api-75bd89fdfc-7n9z4\" (UID: \"b7ac2673-e6fe-4c31-b278-64345475da61\") " pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" Oct 31 13:06:35.255269 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.255186 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ac2673-e6fe-4c31-b278-64345475da61-tls-certs\") pod \"model-serving-api-75bd89fdfc-7n9z4\" (UID: \"b7ac2673-e6fe-4c31-b278-64345475da61\") " pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" Oct 31 13:06:35.356588 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.356555 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk4jw\" (UniqueName: \"kubernetes.io/projected/b7ac2673-e6fe-4c31-b278-64345475da61-kube-api-access-hk4jw\") pod \"model-serving-api-75bd89fdfc-7n9z4\" (UID: \"b7ac2673-e6fe-4c31-b278-64345475da61\") " pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" Oct 31 13:06:35.356734 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.356594 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ac2673-e6fe-4c31-b278-64345475da61-tls-certs\") pod \"model-serving-api-75bd89fdfc-7n9z4\" (UID: \"b7ac2673-e6fe-4c31-b278-64345475da61\") " pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" Oct 31 13:06:35.358954 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.358930 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ac2673-e6fe-4c31-b278-64345475da61-tls-certs\") pod \"model-serving-api-75bd89fdfc-7n9z4\" (UID: \"b7ac2673-e6fe-4c31-b278-64345475da61\") " pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" Oct 31 13:06:35.370721 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.370693 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk4jw\" (UniqueName: \"kubernetes.io/projected/b7ac2673-e6fe-4c31-b278-64345475da61-kube-api-access-hk4jw\") pod \"model-serving-api-75bd89fdfc-7n9z4\" (UID: \"b7ac2673-e6fe-4c31-b278-64345475da61\") " pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" Oct 31 13:06:35.483568 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.483490 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" Oct 31 13:06:35.761306 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.761196 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7n9z4"] Oct 31 13:06:35.764207 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:06:35.764181 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ac2673_e6fe_4c31_b278_64345475da61.slice/crio-4fcc92017c1f6ec7f9f0582c0f66286e5e05f48086a5f89f071e76726e2016dd WatchSource:0}: Error finding container 4fcc92017c1f6ec7f9f0582c0f66286e5e05f48086a5f89f071e76726e2016dd: Status 404 returned error can't find the container with id 4fcc92017c1f6ec7f9f0582c0f66286e5e05f48086a5f89f071e76726e2016dd Oct 31 13:06:35.807141 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:35.807110 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" event={"ID":"b7ac2673-e6fe-4c31-b278-64345475da61","Type":"ContainerStarted","Data":"4fcc92017c1f6ec7f9f0582c0f66286e5e05f48086a5f89f071e76726e2016dd"} Oct 31 13:06:37.816681 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:37.816594 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" event={"ID":"b7ac2673-e6fe-4c31-b278-64345475da61","Type":"ContainerStarted","Data":"cc9222f7a53f3b92fe3e6a3c9e1a7f6953d33a2b2a5e91a1e62fc882c704b4d4"} Oct 31 13:06:37.817074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:37.816687 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" Oct 31 13:06:37.845612 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:37.845563 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" podStartSLOduration=1.173136073 podStartE2EDuration="2.845547156s" totalImagesPullingTime="1.672411083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:37.844944301 +0000 UTC m=+414.690110945" watchObservedRunningTime="2026-10-31 13:06:37.845547156 +0000 UTC m=+414.690713574" Oct 31 13:06:39.784551 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:39.784522 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-hptgn" Oct 31 13:06:48.826218 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:06:48.826190 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-7n9z4" Oct 31 13:06:53.829908 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:06:53.829868 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:07:23.830769 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:07:23.830726 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:07:46.892655 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:46.892619 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:07:53.832280 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:07:53.832238 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:07:56.421975 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.421879 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq"] Oct 31 13:07:56.426415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.426398 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.450507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.450461 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq"] Oct 31 13:07:56.481033 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.481006 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:07:56.481311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.481292 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:07:56.484809 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.484791 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 13:07:56.484894 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.484865 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 13:07:56.484894 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.484885 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2hbvg\"" Oct 31 13:07:56.510392 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.510368 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trc8h\" (UniqueName: \"kubernetes.io/projected/c15373be-9ca7-4874-8cf6-f5187d341e89-kube-api-access-trc8h\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.510471 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.510398 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-metrics-certs\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.510471 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.510423 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-maas-api-tls\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.510617 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.510599 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15373be-9ca7-4874-8cf6-f5187d341e89-openshift-service-ca\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.611861 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.611822 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15373be-9ca7-4874-8cf6-f5187d341e89-openshift-service-ca\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.611861 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.611867 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trc8h\" (UniqueName: \"kubernetes.io/projected/c15373be-9ca7-4874-8cf6-f5187d341e89-kube-api-access-trc8h\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.612065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.611885 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-metrics-certs\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.612065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.611908 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-maas-api-tls\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.612688 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.612660 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15373be-9ca7-4874-8cf6-f5187d341e89-openshift-service-ca\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.614304 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.614284 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-maas-api-tls\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.614445 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.614420 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-metrics-certs\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.625779 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.625754 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trc8h\" (UniqueName: \"kubernetes.io/projected/c15373be-9ca7-4874-8cf6-f5187d341e89-kube-api-access-trc8h\") pod \"maas-api-54d9cc59bd-dlrpq\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:56.738749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:56.738655 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:07:57.016672 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:57.016600 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq"] Oct 31 13:07:57.020077 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:07:57.020047 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15373be_9ca7_4874_8cf6_f5187d341e89.slice/crio-d9e71a8c63ce2f6cac89f6a363305e84ecde5f3a0be63b21222710870353a0bd WatchSource:0}: Error finding container d9e71a8c63ce2f6cac89f6a363305e84ecde5f3a0be63b21222710870353a0bd: Status 404 returned error can't find the container with id d9e71a8c63ce2f6cac89f6a363305e84ecde5f3a0be63b21222710870353a0bd Oct 31 13:07:57.131523 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:07:57.131466 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" event={"ID":"c15373be-9ca7-4874-8cf6-f5187d341e89","Type":"ContainerStarted","Data":"d9e71a8c63ce2f6cac89f6a363305e84ecde5f3a0be63b21222710870353a0bd"} Oct 31 13:08:00.147944 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:00.147910 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" event={"ID":"c15373be-9ca7-4874-8cf6-f5187d341e89","Type":"ContainerStarted","Data":"0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2"} Oct 31 13:08:00.148340 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:00.148016 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:08:00.175375 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:00.175325 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" podStartSLOduration=1.160493072 podStartE2EDuration="4.175311767s" totalImagesPullingTime="3.014818695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:00.1734444 +0000 UTC m=+497.018610819" watchObservedRunningTime="2026-10-31 13:08:00.175311767 +0000 UTC m=+497.020478184" Oct 31 13:08:06.156755 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:06.156726 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:08:23.832111 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:08:23.832073 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:08:53.831693 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:08:53.831653 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:08:54.140717 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:54.140649 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:08:56.008201 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.008172 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:08:56.192289 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.192260 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj"] Oct 31 13:08:56.196366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.196341 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.207564 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.207535 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 13:08:56.207690 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.207554 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 13:08:56.208607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.208587 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 13:08:56.208722 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.208686 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tcfhh\"" Oct 31 13:08:56.210016 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.209992 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj"] Oct 31 13:08:56.279918 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.279859 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.279918 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.279903 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d5dc13e-a93d-4867-bd22-ccf73246c370-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.280052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.280008 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.280052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.280038 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.280125 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.280108 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdkst\" (UniqueName: \"kubernetes.io/projected/1d5dc13e-a93d-4867-bd22-ccf73246c370-kube-api-access-bdkst\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.280158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.280138 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.380740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.380708 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.380740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.380745 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.380968 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.380774 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdkst\" (UniqueName: \"kubernetes.io/projected/1d5dc13e-a93d-4867-bd22-ccf73246c370-kube-api-access-bdkst\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.380968 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.380798 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.380968 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.380817 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.380968 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.380856 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d5dc13e-a93d-4867-bd22-ccf73246c370-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.381164 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.381120 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.381222 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.381184 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.381279 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.381250 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.383106 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.383082 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d5dc13e-a93d-4867-bd22-ccf73246c370-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.383270 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.383254 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d5dc13e-a93d-4867-bd22-ccf73246c370-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.396494 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.396454 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdkst\" (UniqueName: \"kubernetes.io/projected/1d5dc13e-a93d-4867-bd22-ccf73246c370-kube-api-access-bdkst\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj\" (UID: \"1d5dc13e-a93d-4867-bd22-ccf73246c370\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.508247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.508207 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:56.790768 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:56.790744 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj"] Oct 31 13:08:56.792925 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:08:56.792894 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d5dc13e_a93d_4867_bd22_ccf73246c370.slice/crio-5e611d1b2e0603239b22956135ef10cd83ffe76866e5979e041bba0952cfaf33 WatchSource:0}: Error finding container 5e611d1b2e0603239b22956135ef10cd83ffe76866e5979e041bba0952cfaf33: Status 404 returned error can't find the container with id 5e611d1b2e0603239b22956135ef10cd83ffe76866e5979e041bba0952cfaf33 Oct 31 13:08:57.368890 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:57.368829 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" event={"ID":"1d5dc13e-a93d-4867-bd22-ccf73246c370","Type":"ContainerStarted","Data":"5e611d1b2e0603239b22956135ef10cd83ffe76866e5979e041bba0952cfaf33"} Oct 31 13:08:58.375049 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.375012 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" event={"ID":"1d5dc13e-a93d-4867-bd22-ccf73246c370","Type":"ContainerStarted","Data":"4ec5669469cedc99fd0eb20e656ab5945ed4b14df976ed133fec3680fab90c41"} Oct 31 13:08:58.404016 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.403966 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podStartSLOduration=1.2635364230000001 podStartE2EDuration="2.403951505s" totalImagesPullingTime="1.140415082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:58.403157051 +0000 UTC m=+555.248323468" watchObservedRunningTime="2026-10-31 13:08:58.403951505 +0000 UTC m=+555.249117938" Oct 31 13:08:58.509357 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.509309 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:08:58.626329 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.626259 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd"] Oct 31 13:08:58.629620 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.629603 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.639447 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.639430 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 13:08:58.644157 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.644136 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd"] Oct 31 13:08:58.698852 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.698816 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ln9k\" (UniqueName: \"kubernetes.io/projected/a4599d10-3cf5-45df-bb98-b52d2911367b-kube-api-access-7ln9k\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.699030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.698880 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4599d10-3cf5-45df-bb98-b52d2911367b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.699030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.698946 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.699030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.698975 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.699030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.698994 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.699030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.699013 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.799654 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.799621 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.799818 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.799679 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ln9k\" (UniqueName: \"kubernetes.io/projected/a4599d10-3cf5-45df-bb98-b52d2911367b-kube-api-access-7ln9k\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.799818 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.799744 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4599d10-3cf5-45df-bb98-b52d2911367b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.799818 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.799796 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.799968 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.799824 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.799968 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.799850 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.800078 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.800054 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.800163 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.800140 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.800263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.800245 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.802106 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.802088 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4599d10-3cf5-45df-bb98-b52d2911367b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.802240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.802224 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4599d10-3cf5-45df-bb98-b52d2911367b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.813169 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.813147 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ln9k\" (UniqueName: \"kubernetes.io/projected/a4599d10-3cf5-45df-bb98-b52d2911367b-kube-api-access-7ln9k\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd\" (UID: \"a4599d10-3cf5-45df-bb98-b52d2911367b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:58.943607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:58.943517 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:08:59.039446 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:59.039412 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:08:59.221529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:59.221503 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd"] Oct 31 13:08:59.223907 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:08:59.223880 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4599d10_3cf5_45df_bb98_b52d2911367b.slice/crio-43745e25a9ab4d371a0a706c7605d12570523597d8707a717a9cd49f30f515f9 WatchSource:0}: Error finding container 43745e25a9ab4d371a0a706c7605d12570523597d8707a717a9cd49f30f515f9: Status 404 returned error can't find the container with id 43745e25a9ab4d371a0a706c7605d12570523597d8707a717a9cd49f30f515f9 Oct 31 13:08:59.380913 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:08:59.380880 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" event={"ID":"a4599d10-3cf5-45df-bb98-b52d2911367b","Type":"ContainerStarted","Data":"43745e25a9ab4d371a0a706c7605d12570523597d8707a717a9cd49f30f515f9"} Oct 31 13:09:00.104151 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.104113 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:09:00.387253 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.387156 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" event={"ID":"a4599d10-3cf5-45df-bb98-b52d2911367b","Type":"ContainerStarted","Data":"6f9a336b6f8ca2d578f938938b295189acd0dab8bcf82d49fdc652ef259c5c97"} Oct 31 13:09:00.412307 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.412264 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" podStartSLOduration=2.100551939 podStartE2EDuration="2.412251571s" totalImagesPullingTime="311.699632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:00.411040306 +0000 UTC m=+557.256206724" watchObservedRunningTime="2026-10-31 13:09:00.412251571 +0000 UTC m=+557.257417986" Oct 31 13:09:00.614547 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.614510 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg"] Oct 31 13:09:00.618072 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.618051 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.622543 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.622522 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 13:09:00.630489 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.630449 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg"] Oct 31 13:09:00.714268 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.714192 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.714268 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.714244 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda89da9-cda2-4148-ae7b-70f8966b3769-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.714521 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.714314 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.714521 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.714358 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.714521 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.714416 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.714521 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.714439 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgxn\" (UniqueName: \"kubernetes.io/projected/fda89da9-cda2-4148-ae7b-70f8966b3769-kube-api-access-bwgxn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.815016 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.814976 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda89da9-cda2-4148-ae7b-70f8966b3769-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.815160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.815024 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.815160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.815060 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.815160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.815077 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.815160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.815103 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgxn\" (UniqueName: \"kubernetes.io/projected/fda89da9-cda2-4148-ae7b-70f8966b3769-kube-api-access-bwgxn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.815160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.815125 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.815576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.815550 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.815655 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.815556 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.815655 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.815598 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.817401 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.817371 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda89da9-cda2-4148-ae7b-70f8966b3769-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.817645 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.817628 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda89da9-cda2-4148-ae7b-70f8966b3769-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.833493 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.833448 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgxn\" (UniqueName: \"kubernetes.io/projected/fda89da9-cda2-4148-ae7b-70f8966b3769-kube-api-access-bwgxn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xtnsg\" (UID: \"fda89da9-cda2-4148-ae7b-70f8966b3769\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.932584 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.932546 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:00.944664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:00.944636 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:09:01.249001 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:01.248975 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg"] Oct 31 13:09:01.250956 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:09:01.250932 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda89da9_cda2_4148_ae7b_70f8966b3769.slice/crio-16ce8c22bce73c560320971e88a3f935e3f174c06c5aeb6f5ef56c82e8781177 WatchSource:0}: Error finding container 16ce8c22bce73c560320971e88a3f935e3f174c06c5aeb6f5ef56c82e8781177: Status 404 returned error can't find the container with id 16ce8c22bce73c560320971e88a3f935e3f174c06c5aeb6f5ef56c82e8781177 Oct 31 13:09:01.392069 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:01.392034 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" event={"ID":"fda89da9-cda2-4148-ae7b-70f8966b3769","Type":"ContainerStarted","Data":"16ce8c22bce73c560320971e88a3f935e3f174c06c5aeb6f5ef56c82e8781177"} Oct 31 13:09:02.397249 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:02.397207 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" event={"ID":"fda89da9-cda2-4148-ae7b-70f8966b3769","Type":"ContainerStarted","Data":"87bd2b081364d9d98937b00b13413bebd1c26640dfc9b865b1450222aa3a5468"} Oct 31 13:09:02.427019 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:02.426971 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" podStartSLOduration=2.14367702 podStartE2EDuration="2.426956768s" totalImagesPullingTime="283.279748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:02.426363969 +0000 UTC m=+559.271530390" watchObservedRunningTime="2026-10-31 13:09:02.426956768 +0000 UTC m=+559.272123188" Oct 31 13:09:02.932837 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:02.932797 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:03.369608 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.369567 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg"] Oct 31 13:09:03.372971 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.372952 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.388343 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.388321 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 13:09:03.388554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.388531 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg"] Oct 31 13:09:03.434781 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.434753 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8269080b-1265-480e-a205-976decbd5bfa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.434781 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.434783 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbb8f\" (UniqueName: \"kubernetes.io/projected/8269080b-1265-480e-a205-976decbd5bfa-kube-api-access-pbb8f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.435212 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.434802 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.435212 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.435014 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.435212 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.435046 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.435212 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.435078 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.536209 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.536175 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.536209 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.536212 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.536405 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.536230 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.536405 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.536271 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8269080b-1265-480e-a205-976decbd5bfa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.536405 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.536288 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbb8f\" (UniqueName: \"kubernetes.io/projected/8269080b-1265-480e-a205-976decbd5bfa-kube-api-access-pbb8f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.536405 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.536306 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.536689 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.536665 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.536689 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.536677 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.536806 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.536746 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.538511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.538489 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8269080b-1265-480e-a205-976decbd5bfa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.538839 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.538823 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8269080b-1265-480e-a205-976decbd5bfa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.555383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.555357 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbb8f\" (UniqueName: \"kubernetes.io/projected/8269080b-1265-480e-a205-976decbd5bfa-kube-api-access-pbb8f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg\" (UID: \"8269080b-1265-480e-a205-976decbd5bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.685673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.685588 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:03.976668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:03.976593 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg"] Oct 31 13:09:03.981057 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:09:03.981029 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8269080b_1265_480e_a205_976decbd5bfa.slice/crio-357dd17afa961f8d53c60e1574cc7d1054b507da5efd5a78ae21c5373fef7391 WatchSource:0}: Error finding container 357dd17afa961f8d53c60e1574cc7d1054b507da5efd5a78ae21c5373fef7391: Status 404 returned error can't find the container with id 357dd17afa961f8d53c60e1574cc7d1054b507da5efd5a78ae21c5373fef7391 Oct 31 13:09:04.407072 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:04.407032 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" event={"ID":"8269080b-1265-480e-a205-976decbd5bfa","Type":"ContainerStarted","Data":"dd121354ffbdf2352a69f42c48254fb47843eb2afb1bd44e6643533b81ab5d9c"} Oct 31 13:09:04.407237 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:04.407104 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" event={"ID":"8269080b-1265-480e-a205-976decbd5bfa","Type":"ContainerStarted","Data":"357dd17afa961f8d53c60e1574cc7d1054b507da5efd5a78ae21c5373fef7391"} Oct 31 13:09:04.432154 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:04.432108 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" podStartSLOduration=1.118394619 podStartE2EDuration="1.432095363s" totalImagesPullingTime="313.700744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:04.430629436 +0000 UTC m=+561.275795853" watchObservedRunningTime="2026-10-31 13:09:04.432095363 +0000 UTC m=+561.277261780" Oct 31 13:09:04.685786 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:04.685754 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:06.321173 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:06.321126 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:09:06.509144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:06.509106 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:09:06.521705 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:06.521676 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:09:07.431339 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:07.431313 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:09:08.944146 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:08.944110 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:09:08.956900 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:08.956879 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:09:09.440464 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:09.440437 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd" Oct 31 13:09:10.933160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:10.933111 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:10.945719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:10.945689 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:11.446885 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:11.446856 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" Oct 31 13:09:13.686594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:13.686561 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:13.699222 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:13.699194 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:14.459245 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:14.459212 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" Oct 31 13:09:23.831679 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:09:23.831575 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:09:25.177379 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.177325 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm"] Oct 31 13:09:25.180605 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.180587 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.193196 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.193174 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 13:09:25.196871 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.196852 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dmjbc\"" Oct 31 13:09:25.219630 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.219606 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm"] Oct 31 13:09:25.329248 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.329214 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.329248 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.329247 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsbnl\" (UniqueName: \"kubernetes.io/projected/ce755ffc-b2d4-45c2-9054-6022e18db453-kube-api-access-xsbnl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.329445 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.329343 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.329445 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.329399 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce755ffc-b2d4-45c2-9054-6022e18db453-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.329561 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.329472 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.329561 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.329521 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.329561 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.329556 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.329699 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.329613 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.329699 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.329634 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.430992 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.430898 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.430992 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.430933 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsbnl\" (UniqueName: \"kubernetes.io/projected/ce755ffc-b2d4-45c2-9054-6022e18db453-kube-api-access-xsbnl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.430992 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.430971 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.431282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.431007 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce755ffc-b2d4-45c2-9054-6022e18db453-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.431282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.431045 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.431282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.431061 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.431282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.431181 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.431282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.431224 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.431282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.431250 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.432160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.432128 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.432378 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.432169 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.433534 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.432726 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce755ffc-b2d4-45c2-9054-6022e18db453-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.433534 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.432951 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.433534 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.433224 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.436085 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.436057 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.441521 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.441497 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.455493 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.455453 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce755ffc-b2d4-45c2-9054-6022e18db453-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.458328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.458303 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsbnl\" (UniqueName: \"kubernetes.io/projected/ce755ffc-b2d4-45c2-9054-6022e18db453-kube-api-access-xsbnl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vl2sm\" (UID: \"ce755ffc-b2d4-45c2-9054-6022e18db453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.493962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.493927 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:25.781072 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:25.781048 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm"] Oct 31 13:09:25.782887 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:09:25.782843 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce755ffc_b2d4_45c2_9054_6022e18db453.slice/crio-c13760be1530c677eebfa468258ead64a49fd7f20ee1682f8f7996f517c9ed8c WatchSource:0}: Error finding container c13760be1530c677eebfa468258ead64a49fd7f20ee1682f8f7996f517c9ed8c: Status 404 returned error can't find the container with id c13760be1530c677eebfa468258ead64a49fd7f20ee1682f8f7996f517c9ed8c Oct 31 13:09:26.499131 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:26.499065 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" event={"ID":"ce755ffc-b2d4-45c2-9054-6022e18db453","Type":"ContainerStarted","Data":"c13760be1530c677eebfa468258ead64a49fd7f20ee1682f8f7996f517c9ed8c"} Oct 31 13:09:38.186400 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:38.186367 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:09:38.186668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:38.186439 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:09:38.186668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:38.186467 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:09:38.551093 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:38.551014 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" event={"ID":"ce755ffc-b2d4-45c2-9054-6022e18db453","Type":"ContainerStarted","Data":"82993b715a1d7beb1c71f60a83a1b57d6dbd3c82f6620c0aaf4e3d3b0c341dfe"} Oct 31 13:09:38.577770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:38.577718 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" podStartSLOduration=1.176955456 podStartE2EDuration="13.577702641s" totalImagesPullingTime="12.400747185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:38.576673373 +0000 UTC m=+595.421839790" watchObservedRunningTime="2026-10-31 13:09:38.577702641 +0000 UTC m=+595.422869058" Oct 31 13:09:39.494931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:39.494900 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:39.496556 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:39.496522 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vl2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 13:09:39.496693 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:39.496574 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" podUID="ce755ffc-b2d4-45c2-9054-6022e18db453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 13:09:40.200380 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.200344 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc"] Oct 31 13:09:40.203949 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.203932 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.224512 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.221679 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc"] Oct 31 13:09:40.374956 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.374916 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.374956 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.374964 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.375186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.374981 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.375186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.375112 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xhj9\" (UniqueName: \"kubernetes.io/projected/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-kube-api-access-7xhj9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.375186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.375142 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.375186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.375163 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.375352 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.375264 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.375352 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.375310 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.375352 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.375329 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476295 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476214 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476295 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476250 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476295 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476266 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476295 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476298 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xhj9\" (UniqueName: \"kubernetes.io/projected/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-kube-api-access-7xhj9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476320 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476339 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476382 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476420 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476447 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476697 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476764 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.476862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.476838 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.477112 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.477088 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.477269 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.477252 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.478774 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.478754 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.478962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.478942 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.489966 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.489940 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.494660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.494641 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:40.494912 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.494886 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vl2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 13:09:40.494996 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.494939 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" podUID="ce755ffc-b2d4-45c2-9054-6022e18db453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 13:09:40.495263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.495244 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xhj9\" (UniqueName: \"kubernetes.io/projected/9ec44fe2-a89f-4ae6-8f6f-7220557b19b1-kube-api-access-7xhj9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nq8qc\" (UID: \"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.515986 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.515964 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:40.829040 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.829008 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc"] Oct 31 13:09:40.832321 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:09:40.832292 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec44fe2_a89f_4ae6_8f6f_7220557b19b1.slice/crio-119eda3d141f3db2587b45d7ae330d1a2a61a5699aa7bb3c9eb8fedbd7d67317 WatchSource:0}: Error finding container 119eda3d141f3db2587b45d7ae330d1a2a61a5699aa7bb3c9eb8fedbd7d67317: Status 404 returned error can't find the container with id 119eda3d141f3db2587b45d7ae330d1a2a61a5699aa7bb3c9eb8fedbd7d67317 Oct 31 13:09:40.835339 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.835307 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:09:40.835403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.835376 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:09:40.835441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:40.835406 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:09:41.494927 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:41.494889 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vl2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 13:09:41.495141 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:41.494958 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" podUID="ce755ffc-b2d4-45c2-9054-6022e18db453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 13:09:41.592117 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:41.592078 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" event={"ID":"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1","Type":"ContainerStarted","Data":"39897d1cb858b24c3cee42c8bfa778e16c4b37fd284f37662d9ab7359036dd16"} Oct 31 13:09:41.592323 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:41.592144 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" event={"ID":"9ec44fe2-a89f-4ae6-8f6f-7220557b19b1","Type":"ContainerStarted","Data":"119eda3d141f3db2587b45d7ae330d1a2a61a5699aa7bb3c9eb8fedbd7d67317"} Oct 31 13:09:41.622219 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:41.622169 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" podStartSLOduration=1.622154592 podStartE2EDuration="1.622154592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:41.620253044 +0000 UTC m=+598.465419462" watchObservedRunningTime="2026-10-31 13:09:41.622154592 +0000 UTC m=+598.467321010" Oct 31 13:09:42.495047 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:42.495002 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vl2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 13:09:42.495230 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:42.495062 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" podUID="ce755ffc-b2d4-45c2-9054-6022e18db453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 13:09:42.516584 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:42.516550 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:42.518010 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:42.517986 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nq8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 13:09:42.518116 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:42.518031 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" podUID="9ec44fe2-a89f-4ae6-8f6f-7220557b19b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 13:09:43.324426 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:43.324397 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:09:43.324723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:43.324610 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:09:43.494732 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:43.494692 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vl2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 13:09:43.495184 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:43.495138 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" podUID="ce755ffc-b2d4-45c2-9054-6022e18db453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 13:09:43.516860 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:43.516836 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nq8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 13:09:43.517264 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:43.517241 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" podUID="9ec44fe2-a89f-4ae6-8f6f-7220557b19b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 13:09:44.494422 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:44.494390 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vl2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 13:09:44.494583 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:44.494441 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" podUID="ce755ffc-b2d4-45c2-9054-6022e18db453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 13:09:44.516245 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:44.516211 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nq8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 13:09:44.516397 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:44.516259 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" podUID="9ec44fe2-a89f-4ae6-8f6f-7220557b19b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 13:09:45.494463 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:45.494432 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vl2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 13:09:45.494880 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:45.494494 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" podUID="ce755ffc-b2d4-45c2-9054-6022e18db453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 13:09:45.516555 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:45.516533 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nq8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 13:09:45.516626 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:45.516577 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" podUID="9ec44fe2-a89f-4ae6-8f6f-7220557b19b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 13:09:46.516744 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:46.516693 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nq8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 13:09:46.517279 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:46.516762 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" podUID="9ec44fe2-a89f-4ae6-8f6f-7220557b19b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 13:09:46.846275 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:46.846242 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:46.848467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:46.848440 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" Oct 31 13:09:47.517210 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:47.517169 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nq8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 13:09:47.517633 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:47.517239 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" podUID="9ec44fe2-a89f-4ae6-8f6f-7220557b19b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 13:09:49.130668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:49.130633 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:49.131143 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:49.131033 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:49.131904 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:09:49.131884 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" Oct 31 13:09:53.832438 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:09:53.832398 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:10:21.203529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.203496 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh"] Oct 31 13:10:21.212501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.212466 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.220943 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.220917 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh"] Oct 31 13:10:21.221573 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.221548 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f9e24f-openshift-default-dockercfg-xqbc5\"" Oct 31 13:10:21.298395 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.298357 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf"] Oct 31 13:10:21.302511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.302453 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.316891 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.316866 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf"] Oct 31 13:10:21.319431 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.319410 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-426787-openshift-default-dockercfg-7cnrw\"" Oct 31 13:10:21.344963 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.344933 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.345078 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.344991 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.345119 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.345079 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.345192 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.345179 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.345229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.345203 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.345263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.345237 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0d0c59-d655-4733-993c-d89bca07faf9-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.345309 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.345257 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.345348 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.345305 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.345423 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.345411 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxpbs\" (UniqueName: \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-kube-api-access-jxpbs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.405285 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.405257 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf"] Oct 31 13:10:21.408706 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.408687 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.417215 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.417195 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2a262a-openshift-default-dockercfg-bjmvk\"" Oct 31 13:10:21.424516 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.423889 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf"] Oct 31 13:10:21.446524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446500 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.446723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446535 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.446723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446574 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0d0c59-d655-4733-993c-d89bca07faf9-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.446723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446672 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.446906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446735 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.446906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446765 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.446906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446839 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.446906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446863 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.446906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446892 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.447318 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446924 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.447318 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446948 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.447318 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.446958 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.447318 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.447119 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.447318 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.447118 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.447318 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.447181 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxpbs\" (UniqueName: \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-kube-api-access-jxpbs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.447318 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.447270 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.447952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.447336 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.447952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.447271 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.447952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.447388 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.447952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.447635 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.447952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.447668 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k579r\" (UniqueName: \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-kube-api-access-k579r\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.448236 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.448040 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.448236 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.448058 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0d0c59-d655-4733-993c-d89bca07faf9-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.449229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.449209 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.450130 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.450111 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.465611 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.465563 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxpbs\" (UniqueName: \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-kube-api-access-jxpbs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.466135 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.466114 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.525103 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.525075 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs"] Oct 31 13:10:21.527712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.527687 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:21.530128 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.530111 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.534942 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.534925 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a2c006-openshift-default-dockercfg-shg42\"" Oct 31 13:10:21.543285 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.543263 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs"] Oct 31 13:10:21.549373 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549346 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.549540 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549387 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.549540 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549421 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.549652 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549604 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5jl\" (UniqueName: \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-kube-api-access-2w5jl\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.549707 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549690 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.549748 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549733 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.549791 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549763 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.549829 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549795 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.549829 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549822 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7eb1ab-7542-44aa-a93d-d3186844631a-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.549902 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549852 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.549902 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549879 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.549976 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549904 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.549976 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549937 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.550050 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.549971 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.550050 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.550025 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.550142 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.550052 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.550142 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.550078 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k579r\" (UniqueName: \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-kube-api-access-k579r\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.550142 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.550120 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.550837 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.550812 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.551151 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.551125 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.551265 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.551170 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.551407 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.551390 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.552469 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.552036 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.553820 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.553795 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.553995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.553977 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.564620 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.564599 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.564718 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.564678 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k579r\" (UniqueName: \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-kube-api-access-k579r\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.621567 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.621535 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:21.651015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.650982 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xgq\" (UniqueName: \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-kube-api-access-z7xgq\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.651167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651055 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.651228 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651157 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.651228 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651204 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7eb1ab-7542-44aa-a93d-d3186844631a-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.651311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651237 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.651311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651266 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.651311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651293 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651338 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651386 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651456 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651519 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651554 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651586 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651624 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651649 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651683 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651721 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.651878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.651776 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5jl\" (UniqueName: \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-kube-api-access-2w5jl\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.653212 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.653142 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7eb1ab-7542-44aa-a93d-d3186844631a-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.653591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.653563 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.653738 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.653714 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.653808 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.653720 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.653868 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.653843 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.656075 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.656053 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.657507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.657462 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.665243 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.665209 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5jl\" (UniqueName: \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-kube-api-access-2w5jl\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.665354 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.665334 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.728932 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.728664 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:21.753213 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.753182 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xgq\" (UniqueName: \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-kube-api-access-z7xgq\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.753335 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.753240 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.753335 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.753291 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.753454 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.753335 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.753454 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.753378 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.753454 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.753406 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.753454 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.753435 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.753677 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.753464 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.754273 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.753838 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.754273 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.753881 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.754273 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.754056 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.754273 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.754221 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.754273 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.754233 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.754881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.754856 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.756499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.756458 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.756648 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.756630 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.767501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.767467 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xgq\" (UniqueName: \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-kube-api-access-z7xgq\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.769301 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.769273 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.808987 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.808956 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn"] Oct 31 13:10:21.816078 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.814777 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:21.828166 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.828143 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn"] Oct 31 13:10:21.831103 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.831082 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh"] Oct 31 13:10:21.836571 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.836550 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7bf66c-openshift-default-dockercfg-htxtf\"" Oct 31 13:10:21.841791 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:10:21.841763 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0d0c59_d655_4733_993c_d89bca07faf9.slice/crio-7edbf413a75e833af0415f7f146bfe013c2f70fae4576dd0ca284088439e4335 WatchSource:0}: Error finding container 7edbf413a75e833af0415f7f146bfe013c2f70fae4576dd0ca284088439e4335: Status 404 returned error can't find the container with id 7edbf413a75e833af0415f7f146bfe013c2f70fae4576dd0ca284088439e4335 Oct 31 13:10:21.844869 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.844837 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:21.845260 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.845239 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:21.845388 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.845373 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:21.867178 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.867156 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:21.914524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.914496 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf"] Oct 31 13:10:21.916347 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:10:21.916321 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe40c82_7a02_4fa3_8987_3f88fdf3f60f.slice/crio-c9d1aba92ba1bf89412ad9a496b0c876254a0a385691075cd4eab4923f7bf887 WatchSource:0}: Error finding container c9d1aba92ba1bf89412ad9a496b0c876254a0a385691075cd4eab4923f7bf887: Status 404 returned error can't find the container with id c9d1aba92ba1bf89412ad9a496b0c876254a0a385691075cd4eab4923f7bf887 Oct 31 13:10:21.918686 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.918650 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:21.918768 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.918756 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:21.918854 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.918801 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:21.955355 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.955322 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:21.955519 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.955391 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmvl\" (UniqueName: \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-kube-api-access-qlmvl\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:21.955519 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.955436 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707ba835-82ff-49ae-b982-4f75ecce68ab-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:21.958175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.956760 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:21.958175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.957341 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:21.958175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.957933 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:21.958175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.957980 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:21.958175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.958002 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:21.958175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:21.958025 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.030492 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.030441 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf"] Oct 31 13:10:22.042394 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:10:22.042357 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7eb1ab_7542_44aa_a93d_d3186844631a.slice/crio-fde6b3a01fd91b868b556e6dcc733fd6abc0755e042a3c144a71be37dd37e6e2 WatchSource:0}: Error finding container fde6b3a01fd91b868b556e6dcc733fd6abc0755e042a3c144a71be37dd37e6e2: Status 404 returned error can't find the container with id fde6b3a01fd91b868b556e6dcc733fd6abc0755e042a3c144a71be37dd37e6e2 Oct 31 13:10:22.045613 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.045574 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:22.045721 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.045661 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:22.045721 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.045700 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.064212 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.064273 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmvl\" (UniqueName: \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-kube-api-access-qlmvl\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.064301 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707ba835-82ff-49ae-b982-4f75ecce68ab-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.064331 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.064349 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.064388 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.064415 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.064428 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.064444 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.065021 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.065756 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.065847 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066422 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.066072 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.066883 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.066855 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707ba835-82ff-49ae-b982-4f75ecce68ab-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.068492 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.068441 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.069065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.069044 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.081847 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.081820 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmvl\" (UniqueName: \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-kube-api-access-qlmvl\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.083390 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.083363 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.134948 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.134922 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:22.158750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.158720 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs"] Oct 31 13:10:22.164251 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:10:22.164205 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7c3b8b_6f23_4fe9_be0e_42b98613bcee.slice/crio-4d5b2bd1c72cc0ced75e0c796c86967e7208fa961356ee2b8d74fc4878ca946d WatchSource:0}: Error finding container 4d5b2bd1c72cc0ced75e0c796c86967e7208fa961356ee2b8d74fc4878ca946d: Status 404 returned error can't find the container with id 4d5b2bd1c72cc0ced75e0c796c86967e7208fa961356ee2b8d74fc4878ca946d Oct 31 13:10:22.170536 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.170171 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:22.170536 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.170256 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:22.170536 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.170297 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:22.458594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.458552 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn"] Oct 31 13:10:22.466109 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:10:22.466078 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707ba835_82ff_49ae_b982_4f75ecce68ab.slice/crio-50eff0dc565bf461f344786c2b07fc5bddd34801dd32cf06888f0f1b4ca2dbe9 WatchSource:0}: Error finding container 50eff0dc565bf461f344786c2b07fc5bddd34801dd32cf06888f0f1b4ca2dbe9: Status 404 returned error can't find the container with id 50eff0dc565bf461f344786c2b07fc5bddd34801dd32cf06888f0f1b4ca2dbe9 Oct 31 13:10:22.469378 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.469339 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:22.469457 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.469428 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:22.469519 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.469500 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:22.784591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.784498 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" event={"ID":"707ba835-82ff-49ae-b982-4f75ecce68ab","Type":"ContainerStarted","Data":"dd96b181b7fc7c5e8aed552811fff513274df19cfe4c640981ee5d438d104d9d"} Oct 31 13:10:22.784753 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.784600 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" event={"ID":"707ba835-82ff-49ae-b982-4f75ecce68ab","Type":"ContainerStarted","Data":"50eff0dc565bf461f344786c2b07fc5bddd34801dd32cf06888f0f1b4ca2dbe9"} Oct 31 13:10:22.786160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.786134 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" event={"ID":"ab0d0c59-d655-4733-993c-d89bca07faf9","Type":"ContainerStarted","Data":"6844ab034a05f4a8df5cb0d0595f8fd00bbd52c415fe6ad001d618f185e7a75f"} Oct 31 13:10:22.786279 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.786184 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" event={"ID":"ab0d0c59-d655-4733-993c-d89bca07faf9","Type":"ContainerStarted","Data":"7edbf413a75e833af0415f7f146bfe013c2f70fae4576dd0ca284088439e4335"} Oct 31 13:10:22.787888 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.787848 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" event={"ID":"5c7eb1ab-7542-44aa-a93d-d3186844631a","Type":"ContainerStarted","Data":"f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7"} Oct 31 13:10:22.788003 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.787906 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" event={"ID":"5c7eb1ab-7542-44aa-a93d-d3186844631a","Type":"ContainerStarted","Data":"fde6b3a01fd91b868b556e6dcc733fd6abc0755e042a3c144a71be37dd37e6e2"} Oct 31 13:10:22.790028 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.790001 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" event={"ID":"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f","Type":"ContainerStarted","Data":"d10957a4dc081b7d897ab00aecb54c4036149d642ccc8007213b0458f725f1fc"} Oct 31 13:10:22.790146 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.790052 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" event={"ID":"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f","Type":"ContainerStarted","Data":"c9d1aba92ba1bf89412ad9a496b0c876254a0a385691075cd4eab4923f7bf887"} Oct 31 13:10:22.792355 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.792334 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" event={"ID":"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee","Type":"ContainerStarted","Data":"fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6"} Oct 31 13:10:22.792441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.792368 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" event={"ID":"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee","Type":"ContainerStarted","Data":"4d5b2bd1c72cc0ced75e0c796c86967e7208fa961356ee2b8d74fc4878ca946d"} Oct 31 13:10:22.817816 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.817768 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podStartSLOduration=1.817754811 podStartE2EDuration="1.817754811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:22.816728227 +0000 UTC m=+639.661894672" watchObservedRunningTime="2026-10-31 13:10:22.817754811 +0000 UTC m=+639.662921228" Oct 31 13:10:22.841795 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.841750 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podStartSLOduration=1.841737142 podStartE2EDuration="1.841737142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:22.840405908 +0000 UTC m=+639.685572327" watchObservedRunningTime="2026-10-31 13:10:22.841737142 +0000 UTC m=+639.686903560" Oct 31 13:10:22.866086 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.866037 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" podStartSLOduration=1.8660226450000001 podStartE2EDuration="1.866022645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:22.864146226 +0000 UTC m=+639.709312673" watchObservedRunningTime="2026-10-31 13:10:22.866022645 +0000 UTC m=+639.711189057" Oct 31 13:10:22.867788 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.867767 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:22.888753 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.888716 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podStartSLOduration=1.8887031140000001 podStartE2EDuration="1.888703114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:22.887507792 +0000 UTC m=+639.732674210" watchObservedRunningTime="2026-10-31 13:10:22.888703114 +0000 UTC m=+639.733869532" Oct 31 13:10:22.913272 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:22.913232 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podStartSLOduration=1.9132193389999999 podStartE2EDuration="1.913219339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:22.911995778 +0000 UTC m=+639.757162210" watchObservedRunningTime="2026-10-31 13:10:22.913219339 +0000 UTC m=+639.758385756" Oct 31 13:10:23.135353 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.135325 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:23.140208 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.140187 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:23.528586 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.528505 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:23.533400 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.533376 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:23.622156 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.622131 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:23.626720 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.626699 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:23.729767 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.729739 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:23.734550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.734528 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:23.797237 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.797138 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:23.797237 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.797195 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:23.797926 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.797646 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:23.797926 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.797693 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:23.798109 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.797983 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:23.798109 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.798041 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:23.798211 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.798195 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:23.798544 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.798530 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:23.832350 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:10:23.832317 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:10:23.873973 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:23.873946 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:24.805496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:24.805438 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:24.806549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:24.806530 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:30.499040 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.499008 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:10:30.833098 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.833064 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn"] Oct 31 13:10:30.836660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.836641 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.855693 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.855669 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-a2c006-serving-cert\"" Oct 31 13:10:30.857043 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.857025 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn"] Oct 31 13:10:30.861196 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.861171 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.861313 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.861223 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a1293a-a69b-4a71-95e2-265a03fd633b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.861396 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.861370 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-metrics-certs\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.861513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.861492 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bl59\" (UniqueName: \"kubernetes.io/projected/d3a1293a-a69b-4a71-95e2-265a03fd633b-kube-api-access-7bl59\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.964956 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.964660 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.966026 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.966000 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a1293a-a69b-4a71-95e2-265a03fd633b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.966245 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.966226 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a1293a-a69b-4a71-95e2-265a03fd633b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.966606 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.966587 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-metrics-certs\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.967636 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.967616 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bl59\" (UniqueName: \"kubernetes.io/projected/d3a1293a-a69b-4a71-95e2-265a03fd633b-kube-api-access-7bl59\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.986672 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.986638 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bl59\" (UniqueName: \"kubernetes.io/projected/d3a1293a-a69b-4a71-95e2-265a03fd633b-kube-api-access-7bl59\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.988038 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.987983 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-metrics-certs\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:30.988640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:30.988619 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:31.149728 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.149644 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:31.323080 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.323052 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx"] Oct 31 13:10:31.326676 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.326657 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:10:31.335818 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.335790 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-79c6ff\"" Oct 31 13:10:31.338529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.338502 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx"] Oct 31 13:10:31.371570 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.371544 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f742af-293a-40d6-9a93-75e3030306b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx\" (UID: \"c3f742af-293a-40d6-9a93-75e3030306b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:10:31.371720 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.371631 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptbj\" (UniqueName: \"kubernetes.io/projected/c3f742af-293a-40d6-9a93-75e3030306b8-kube-api-access-8ptbj\") pod \"payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx\" (UID: \"c3f742af-293a-40d6-9a93-75e3030306b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:10:31.446940 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.446784 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn"] Oct 31 13:10:31.450649 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:10:31.450616 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a1293a_a69b_4a71_95e2_265a03fd633b.slice/crio-3a5c9d7b3462a86fc4e39b02ebdafb6533b726842cb562ebaf615f9d64ad71c7 WatchSource:0}: Error finding container 3a5c9d7b3462a86fc4e39b02ebdafb6533b726842cb562ebaf615f9d64ad71c7: Status 404 returned error can't find the container with id 3a5c9d7b3462a86fc4e39b02ebdafb6533b726842cb562ebaf615f9d64ad71c7 Oct 31 13:10:31.452621 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.452601 4542 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:10:31.488497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.472493 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptbj\" (UniqueName: \"kubernetes.io/projected/c3f742af-293a-40d6-9a93-75e3030306b8-kube-api-access-8ptbj\") pod \"payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx\" (UID: \"c3f742af-293a-40d6-9a93-75e3030306b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:10:31.488497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.472655 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f742af-293a-40d6-9a93-75e3030306b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx\" (UID: \"c3f742af-293a-40d6-9a93-75e3030306b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:10:31.488497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.473638 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f742af-293a-40d6-9a93-75e3030306b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx\" (UID: \"c3f742af-293a-40d6-9a93-75e3030306b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:10:31.488497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.487944 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptbj\" (UniqueName: \"kubernetes.io/projected/c3f742af-293a-40d6-9a93-75e3030306b8-kube-api-access-8ptbj\") pod \"payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx\" (UID: \"c3f742af-293a-40d6-9a93-75e3030306b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:10:31.643048 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.642697 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:10:31.900244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.900021 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" event={"ID":"d3a1293a-a69b-4a71-95e2-265a03fd633b","Type":"ContainerStarted","Data":"3a5c9d7b3462a86fc4e39b02ebdafb6533b726842cb562ebaf615f9d64ad71c7"} Oct 31 13:10:31.985112 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:31.961738 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx"] Oct 31 13:10:32.003936 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:10:32.003898 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f742af_293a_40d6_9a93_75e3030306b8.slice/crio-cbaa1fb8073973c3de8764265c88cd475d956f1dcc01e9f2019b4bb1266347cb WatchSource:0}: Error finding container cbaa1fb8073973c3de8764265c88cd475d956f1dcc01e9f2019b4bb1266347cb: Status 404 returned error can't find the container with id cbaa1fb8073973c3de8764265c88cd475d956f1dcc01e9f2019b4bb1266347cb Oct 31 13:10:32.909509 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:32.909394 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" event={"ID":"c3f742af-293a-40d6-9a93-75e3030306b8","Type":"ContainerStarted","Data":"cbaa1fb8073973c3de8764265c88cd475d956f1dcc01e9f2019b4bb1266347cb"} Oct 31 13:10:32.911273 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:32.911246 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" event={"ID":"d3a1293a-a69b-4a71-95e2-265a03fd633b","Type":"ContainerStarted","Data":"6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073"} Oct 31 13:10:32.911415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:32.911378 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:32.936069 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:32.936014 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" podStartSLOduration=2.474743974 podStartE2EDuration="2.935996937s" totalImagesPullingTime="461.252963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:32.932030703 +0000 UTC m=+649.777197121" watchObservedRunningTime="2026-10-31 13:10:32.935996937 +0000 UTC m=+649.781163393" Oct 31 13:10:33.810889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:33.810853 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk"] Oct 31 13:10:33.815063 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:33.815038 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:33.827546 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:33.825520 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-004f64-serving-cert\"" Oct 31 13:10:33.828077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:33.828053 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk"] Oct 31 13:10:33.916671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:33.916645 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff621f22-5195-4616-9fb0-3604409740e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:33.917119 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:33.917100 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:33.917272 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:33.917259 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-metrics-certs\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:33.917400 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:33.917386 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbr9x\" (UniqueName: \"kubernetes.io/projected/ff621f22-5195-4616-9fb0-3604409740e7-kube-api-access-xbr9x\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:34.022769 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.019607 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff621f22-5195-4616-9fb0-3604409740e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:34.022769 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.019685 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:34.022769 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.019781 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-metrics-certs\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:34.022769 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.019867 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbr9x\" (UniqueName: \"kubernetes.io/projected/ff621f22-5195-4616-9fb0-3604409740e7-kube-api-access-xbr9x\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:34.022769 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.021111 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff621f22-5195-4616-9fb0-3604409740e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:34.027220 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.027148 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-metrics-certs\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:34.028619 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.028453 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:34.036541 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.036506 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbr9x\" (UniqueName: \"kubernetes.io/projected/ff621f22-5195-4616-9fb0-3604409740e7-kube-api-access-xbr9x\") pod \"maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:34.131924 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.131888 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:34.535979 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.535950 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk"] Oct 31 13:10:34.537828 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:10:34.537776 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff621f22_5195_4616_9fb0_3604409740e7.slice/crio-af58f701808c3be1a65123db21a155cdf4fc4df4849fab704b70dbf1e5765a74 WatchSource:0}: Error finding container af58f701808c3be1a65123db21a155cdf4fc4df4849fab704b70dbf1e5765a74: Status 404 returned error can't find the container with id af58f701808c3be1a65123db21a155cdf4fc4df4849fab704b70dbf1e5765a74 Oct 31 13:10:34.726535 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.722083 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds"] Oct 31 13:10:34.733374 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.733349 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.804268 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.801933 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-79c6ff-serving-cert\"" Oct 31 13:10:34.814135 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.813268 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds"] Oct 31 13:10:34.833931 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.833897 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-metrics-certs\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.834074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.833974 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdl2\" (UniqueName: \"kubernetes.io/projected/2541cf2f-8571-49cc-995f-4174130110be-kube-api-access-zfdl2\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.834074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.834003 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2541cf2f-8571-49cc-995f-4174130110be-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.834074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.834033 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-maas-api-tls\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.894870 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.894834 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:10:34.925415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.925380 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" event={"ID":"ff621f22-5195-4616-9fb0-3604409740e7","Type":"ContainerStarted","Data":"af58f701808c3be1a65123db21a155cdf4fc4df4849fab704b70dbf1e5765a74"} Oct 31 13:10:34.935120 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.934829 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-metrics-certs\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.935120 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.934916 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdl2\" (UniqueName: \"kubernetes.io/projected/2541cf2f-8571-49cc-995f-4174130110be-kube-api-access-zfdl2\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.935120 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.934953 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2541cf2f-8571-49cc-995f-4174130110be-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.935120 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.934985 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-maas-api-tls\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.935770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.935742 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2541cf2f-8571-49cc-995f-4174130110be-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.964500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.938912 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-metrics-certs\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:34.969036 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:34.968980 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-maas-api-tls\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:35.018902 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:35.018836 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdl2\" (UniqueName: \"kubernetes.io/projected/2541cf2f-8571-49cc-995f-4174130110be-kube-api-access-zfdl2\") pod \"maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:35.134158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:35.134133 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:35.664306 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:35.649892 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds"] Oct 31 13:10:35.735511 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:10:35.734099 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2541cf2f_8571_49cc_995f_4174130110be.slice/crio-04be76536f20f8e2fafb25b9e80a99cb679d2d5160eec69c31c1b47017daf781 WatchSource:0}: Error finding container 04be76536f20f8e2fafb25b9e80a99cb679d2d5160eec69c31c1b47017daf781: Status 404 returned error can't find the container with id 04be76536f20f8e2fafb25b9e80a99cb679d2d5160eec69c31c1b47017daf781 Oct 31 13:10:35.989577 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:35.987419 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" event={"ID":"ff621f22-5195-4616-9fb0-3604409740e7","Type":"ContainerStarted","Data":"2df4812e0f22adf1d00adb8d4f96a016ef6ac8c7e2c36464cdbdc8108c879d1a"} Oct 31 13:10:35.989577 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:35.988423 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:36.007559 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:35.998608 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" event={"ID":"2541cf2f-8571-49cc-995f-4174130110be","Type":"ContainerStarted","Data":"04be76536f20f8e2fafb25b9e80a99cb679d2d5160eec69c31c1b47017daf781"} Oct 31 13:10:36.046579 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:36.044004 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" podStartSLOduration=2.580973506 podStartE2EDuration="3.04398302s" totalImagesPullingTime="463.009514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:36.040764021 +0000 UTC m=+652.885930441" watchObservedRunningTime="2026-10-31 13:10:36.04398302 +0000 UTC m=+652.889149441" Oct 31 13:10:37.071499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:37.070624 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" event={"ID":"2541cf2f-8571-49cc-995f-4174130110be","Type":"ContainerStarted","Data":"ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e"} Oct 31 13:10:37.071499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:37.070704 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:38.925004 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:38.924882 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:10:38.970513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:38.965876 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" podStartSLOduration=4.239435742 podStartE2EDuration="4.949044299s" totalImagesPullingTime="709.608557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:37.112276237 +0000 UTC m=+653.957442656" watchObservedRunningTime="2026-10-31 13:10:38.949044299 +0000 UTC m=+655.794210717" Oct 31 13:10:39.876967 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:39.876614 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:10:39.876967 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:39.876685 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:10:39.916673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:39.916622 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:10:39.916871 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:39.916703 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:10:39.930396 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:39.930362 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:10:42.238038 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:42.237005 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" event={"ID":"c3f742af-293a-40d6-9a93-75e3030306b8","Type":"ContainerStarted","Data":"67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6"} Oct 31 13:10:42.238038 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:42.237932 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:10:42.264116 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:42.260508 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" podStartSLOduration=1.731797024 podStartE2EDuration="11.260468782s" totalImagesPullingTime="9.528671758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:42.25963095 +0000 UTC m=+659.104797382" watchObservedRunningTime="2026-10-31 13:10:42.260468782 +0000 UTC m=+659.105635200" Oct 31 13:10:43.115887 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:43.115132 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:10:44.147295 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:44.147262 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.289404 4542 generic.go:356] "Generic (PLEG): container finished" podID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerID="dd96b181b7fc7c5e8aed552811fff513274df19cfe4c640981ee5d438d104d9d" exitCode=0 Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.291182 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" event={"ID":"707ba835-82ff-49ae-b982-4f75ecce68ab","Type":"ContainerDied","Data":"dd96b181b7fc7c5e8aed552811fff513274df19cfe4c640981ee5d438d104d9d"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.291340 4542 generic.go:356] "Generic (PLEG): container finished" podID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerID="6844ab034a05f4a8df5cb0d0595f8fd00bbd52c415fe6ad001d618f185e7a75f" exitCode=0 Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.291412 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" event={"ID":"ab0d0c59-d655-4733-993c-d89bca07faf9","Type":"ContainerDied","Data":"6844ab034a05f4a8df5cb0d0595f8fd00bbd52c415fe6ad001d618f185e7a75f"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.291645 4542 scope.go:122] "RemoveContainer" containerID="dd96b181b7fc7c5e8aed552811fff513274df19cfe4c640981ee5d438d104d9d" Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.291777 4542 scope.go:122] "RemoveContainer" containerID="6844ab034a05f4a8df5cb0d0595f8fd00bbd52c415fe6ad001d618f185e7a75f" Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.293169 4542 generic.go:356] "Generic (PLEG): container finished" podID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerID="d10957a4dc081b7d897ab00aecb54c4036149d642ccc8007213b0458f725f1fc" exitCode=0 Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.293237 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" event={"ID":"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f","Type":"ContainerDied","Data":"d10957a4dc081b7d897ab00aecb54c4036149d642ccc8007213b0458f725f1fc"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.293236 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.293316 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.293236 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.293381 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.293407 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.293452 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.293575 4542 scope.go:122] "RemoveContainer" containerID="d10957a4dc081b7d897ab00aecb54c4036149d642ccc8007213b0458f725f1fc" Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.294764 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.294823 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:49.395101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.294856 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:49.646604 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:49.646511 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:10:50.333693 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.333660 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" event={"ID":"707ba835-82ff-49ae-b982-4f75ecce68ab","Type":"ContainerStarted","Data":"ea395df10199b8348bb4861a6cf9bce03d01fae5924ccdec6f29b0077faf1db4"} Oct 31 13:10:50.335376 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.335353 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" event={"ID":"ab0d0c59-d655-4733-993c-d89bca07faf9","Type":"ContainerStarted","Data":"6490e9e5fb0ed1838508baadbfc058e6bab65c52b4717ac54083c77634d387b5"} Oct 31 13:10:50.337441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.336844 4542 generic.go:356] "Generic (PLEG): container finished" podID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerID="f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7" exitCode=0 Oct 31 13:10:50.337441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.336894 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" event={"ID":"5c7eb1ab-7542-44aa-a93d-d3186844631a","Type":"ContainerDied","Data":"f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7"} Oct 31 13:10:50.337441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.337132 4542 scope.go:122] "RemoveContainer" containerID="f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7" Oct 31 13:10:50.343121 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.338355 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:50.343121 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.338439 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:50.343121 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.338520 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:50.395438 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.395399 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" event={"ID":"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f","Type":"ContainerStarted","Data":"39a0dcaf7e4ce535078f9c07bb52ceb47cc891394c50ece4404e70bcceb98b2b"} Oct 31 13:10:50.406940 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.406902 4542 generic.go:356] "Generic (PLEG): container finished" podID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerID="fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6" exitCode=0 Oct 31 13:10:50.407587 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.407453 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" event={"ID":"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee","Type":"ContainerDied","Data":"fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6"} Oct 31 13:10:50.409357 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.409249 4542 scope.go:122] "RemoveContainer" containerID="fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6" Oct 31 13:10:50.416823 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.416776 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:50.417052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.417026 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:50.417224 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.417188 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:10:50.532070 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.531380 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:50.556727 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.546587 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 13:10:50.556727 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.546680 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 13:10:50.645507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.627005 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:50.645507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.627419 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:10:50.645507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:50.627463 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 13:10:51.138565 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.137808 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:51.138565 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.137984 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:10:51.138565 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.138027 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 13:10:51.472455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.450776 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" event={"ID":"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee","Type":"ContainerStarted","Data":"cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70"} Oct 31 13:10:51.498607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.482575 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" event={"ID":"5c7eb1ab-7542-44aa-a93d-d3186844631a","Type":"ContainerStarted","Data":"283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4"} Oct 31 13:10:51.549502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.538072 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 13:10:51.549502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.538132 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 13:10:51.637508 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.636933 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:10:51.637508 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.636995 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 13:10:51.729755 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.729634 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:51.731266 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.731238 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:10:51.731461 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.731438 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 13:10:51.872506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.872033 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:51.872506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.872169 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:10:51.872506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:51.872218 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 13:10:52.139511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:52.138888 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:10:52.139511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:52.138955 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 13:10:52.530560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:52.529841 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 13:10:52.530560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:52.529903 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 13:10:52.624990 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:52.624953 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:10:52.625322 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:52.625273 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 13:10:52.733511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:52.729926 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:10:52.733511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:52.729988 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 13:10:52.868014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:52.867976 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:10:52.868331 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:52.868290 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 13:10:53.138092 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.137991 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:10:53.138352 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.138309 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 13:10:53.535999 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.535894 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 13:10:53.536634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.536606 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 13:10:53.630682 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.630642 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:10:53.631165 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.631134 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 13:10:53.729859 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.728962 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:10:53.729859 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.729018 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 13:10:53.840592 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.830515 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:53.867165 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.841221 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:53.870068 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.867417 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:53.870068 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.867499 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:53.870068 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.869631 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:10:53.870068 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:53.869690 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 13:10:54.146554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.136005 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:10:54.146554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.136081 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 13:10:54.169500 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:10:54.168384 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:10:54.301506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.296674 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:10:54.553549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.536229 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 13:10:54.588613 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.588560 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 13:10:54.624123 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.624082 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:10:54.624426 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.624396 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 13:10:54.809698 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.809590 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:54.889501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.885824 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:10:54.948497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.947791 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 13:10:54.948497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.932212 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:10:54.948497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:54.948195 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 13:10:55.141515 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:55.137855 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:10:55.141515 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:55.137919 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 13:10:55.531951 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:55.531852 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 13:10:55.532527 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:55.532496 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 13:10:55.625938 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:55.625904 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:10:55.626591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:55.626531 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 13:10:55.733971 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:55.733934 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:10:55.734260 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:55.734234 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 13:10:55.869029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:55.868982 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:10:55.869350 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:55.869297 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 13:10:56.139560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:56.138143 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:10:56.139560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:56.138213 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 13:10:56.537600 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:56.531857 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 13:10:56.537600 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:56.531921 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 13:10:56.624017 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:56.623978 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:10:56.624320 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:56.624277 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 13:10:56.731886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:56.731845 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:10:56.732510 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:56.732454 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 13:10:56.874078 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:56.874041 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:10:56.874353 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:56.874323 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 13:10:57.136080 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:57.135979 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:10:57.136351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:57.136309 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 13:10:57.732892 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:57.732510 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:10:57.732892 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:57.732565 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 13:10:57.760503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:57.758536 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:57.763294 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:57.763201 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:10:57.920060 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:57.919408 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:10:57.920060 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:57.919457 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 13:10:58.360952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:58.355795 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:58.427367 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:58.427333 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:10:58.520855 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:58.520825 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:58.523225 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:58.523168 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:10:58.730094 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:58.730012 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:10:58.730094 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:58.730064 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 13:10:58.868013 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:58.867965 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:10:58.868537 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:58.868031 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 13:10:59.399774 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.399736 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg"] Oct 31 13:10:59.406984 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.406962 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.412851 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.412824 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a2c006\"/\"simulator-w1-a2c006-kserve-self-signed-certs\"" Oct 31 13:10:59.412984 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.412920 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a2c006\"/\"default-dockercfg-5bvlg\"" Oct 31 13:10:59.413117 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.413102 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a2c006\"/\"kube-root-ca.crt\"" Oct 31 13:10:59.413198 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.413135 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a2c006\"/\"openshift-service-ca.crt\"" Oct 31 13:10:59.414235 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.414213 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg"] Oct 31 13:10:59.501073 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.501043 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c6f328-8a73-413c-a1a8-cc278e20c6db-tls-certs\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.501244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.501106 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-dshm\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.501244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.501204 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-model-cache\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.501244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.501234 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-tmp-dir\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.501348 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.501264 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-home\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.501385 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.501359 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kqm\" (UniqueName: \"kubernetes.io/projected/01c6f328-8a73-413c-a1a8-cc278e20c6db-kube-api-access-d4kqm\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.602898 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.602860 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-dshm\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.603085 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.602944 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-model-cache\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.603085 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.602980 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-tmp-dir\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.603085 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.603025 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-home\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.603247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.603084 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kqm\" (UniqueName: \"kubernetes.io/projected/01c6f328-8a73-413c-a1a8-cc278e20c6db-kube-api-access-d4kqm\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.603247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.603141 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c6f328-8a73-413c-a1a8-cc278e20c6db-tls-certs\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.603382 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.603357 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-model-cache\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.603448 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.603419 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-home\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.603533 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.603487 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-tmp-dir\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.605507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.605455 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-dshm\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.605732 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.605717 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c6f328-8a73-413c-a1a8-cc278e20c6db-tls-certs\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.617051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.617030 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kqm\" (UniqueName: \"kubernetes.io/projected/01c6f328-8a73-413c-a1a8-cc278e20c6db-kube-api-access-d4kqm\") pod \"simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.720197 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.720117 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:10:59.733768 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.733745 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:59.734951 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.734934 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:10:59.811211 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.811183 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:10:59.872824 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.872797 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:10:59.873878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:10:59.873848 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:11:00.005665 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:00.005641 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg"] Oct 31 13:11:00.007652 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:11:00.007619 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01c6f328_8a73_413c_a1a8_cc278e20c6db.slice/crio-851cf1586ed865bc93aaab7f9cad3619408d7a996634b7c2c41cda1e23c5c5da WatchSource:0}: Error finding container 851cf1586ed865bc93aaab7f9cad3619408d7a996634b7c2c41cda1e23c5c5da: Status 404 returned error can't find the container with id 851cf1586ed865bc93aaab7f9cad3619408d7a996634b7c2c41cda1e23c5c5da Oct 31 13:11:00.713542 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:00.713505 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" event={"ID":"01c6f328-8a73-413c-a1a8-cc278e20c6db","Type":"ContainerStarted","Data":"dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff"} Oct 31 13:11:00.713744 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:00.713553 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" event={"ID":"01c6f328-8a73-413c-a1a8-cc278e20c6db","Type":"ContainerStarted","Data":"851cf1586ed865bc93aaab7f9cad3619408d7a996634b7c2c41cda1e23c5c5da"} Oct 31 13:11:00.720870 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:00.720844 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:11:00.735385 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:00.735345 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podStartSLOduration=1.735332981 podStartE2EDuration="1.735332981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:00.733872141 +0000 UTC m=+677.579038557" watchObservedRunningTime="2026-10-31 13:11:00.735332981 +0000 UTC m=+677.580499399" Oct 31 13:11:08.467073 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.467041 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p"] Oct 31 13:11:08.470916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.470897 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.480185 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.480158 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-79c6ff\"/\"kube-root-ca.crt\"" Oct 31 13:11:08.480327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.480197 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-79c6ff\"/\"openshift-service-ca.crt\"" Oct 31 13:11:08.480327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.480201 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p"] Oct 31 13:11:08.480327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.480322 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-79c6ff\"/\"simulator-w6-79c6ff-kserve-self-signed-certs\"" Oct 31 13:11:08.480450 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.480369 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-79c6ff\"/\"default-dockercfg-rkkdq\"" Oct 31 13:11:08.503225 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.503196 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-model-cache\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.503371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.503284 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tls-certs\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.503441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.503418 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn8b4\" (UniqueName: \"kubernetes.io/projected/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-kube-api-access-cn8b4\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.503508 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.503458 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-home\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.503542 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.503519 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tmp-dir\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.503580 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.503561 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-dshm\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.604957 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.604923 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tls-certs\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.605144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.604992 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn8b4\" (UniqueName: \"kubernetes.io/projected/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-kube-api-access-cn8b4\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.605144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.605018 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-home\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.605144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.605037 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tmp-dir\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.605144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.605058 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-dshm\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.605144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.605101 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-model-cache\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.605504 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.605454 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-home\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.605562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.605539 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tmp-dir\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.605621 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.605602 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-model-cache\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.607345 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.607326 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-dshm\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.607515 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.607495 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tls-certs\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.618550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.618524 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn8b4\" (UniqueName: \"kubernetes.io/projected/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-kube-api-access-cn8b4\") pod \"simulator-w6-79c6ff-kserve-689c5c5d-h9q9p\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:08.784125 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:08.784048 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:09.060582 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:09.060555 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p"] Oct 31 13:11:09.062816 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:11:09.062792 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f116ed8_404c_4dc1_a81d_364a4cbed2bc.slice/crio-77eef2e6fcd560048e3cc58e93b6f7f4b80a8c2cfc9187e2503fb34d6efb8ac6 WatchSource:0}: Error finding container 77eef2e6fcd560048e3cc58e93b6f7f4b80a8c2cfc9187e2503fb34d6efb8ac6: Status 404 returned error can't find the container with id 77eef2e6fcd560048e3cc58e93b6f7f4b80a8c2cfc9187e2503fb34d6efb8ac6 Oct 31 13:11:09.720622 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:09.720591 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:11:09.733114 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:09.733088 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:11:09.757850 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:09.757816 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" event={"ID":"1f116ed8-404c-4dc1-a81d-364a4cbed2bc","Type":"ContainerStarted","Data":"b73524eec74a6cf109978f6d10642882ffa26f38a8be05e8c2113525b3f0c2a9"} Oct 31 13:11:09.757983 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:09.757862 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" event={"ID":"1f116ed8-404c-4dc1-a81d-364a4cbed2bc","Type":"ContainerStarted","Data":"77eef2e6fcd560048e3cc58e93b6f7f4b80a8c2cfc9187e2503fb34d6efb8ac6"} Oct 31 13:11:09.769000 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:09.768976 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:11:09.784124 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:09.784102 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:09.854470 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:09.854411 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podStartSLOduration=1.8543948179999998 podStartE2EDuration="1.854394818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:09.853558238 +0000 UTC m=+686.698724657" watchObservedRunningTime="2026-10-31 13:11:09.854394818 +0000 UTC m=+686.699561237" Oct 31 13:11:11.922466 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:11.922430 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:11:16.781090 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.781059 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn"] Oct 31 13:11:16.784900 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.784883 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.796303 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.796277 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7bf66c\"/\"kube-root-ca.crt\"" Oct 31 13:11:16.796976 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.796948 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7bf66c\"/\"premium-w0-7bf66c-kserve-self-signed-certs\"" Oct 31 13:11:16.797085 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.797044 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7bf66c\"/\"default-dockercfg-nn5dd\"" Oct 31 13:11:16.797144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.797098 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7bf66c\"/\"openshift-service-ca.crt\"" Oct 31 13:11:16.799188 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.799169 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn"] Oct 31 13:11:16.887545 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.887511 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-home\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.887545 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.887544 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tls-certs\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.887800 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.887634 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tmp-dir\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.887800 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.887727 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-model-cache\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.887800 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.887758 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nq29\" (UniqueName: \"kubernetes.io/projected/5c11b2df-1fc1-40ca-86f0-315f19e683e7-kube-api-access-4nq29\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.887956 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.887855 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-dshm\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.988948 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.988915 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-home\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.989129 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.988952 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tls-certs\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.989129 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.989049 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tmp-dir\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.989129 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.989085 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-model-cache\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.989129 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.989104 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nq29\" (UniqueName: \"kubernetes.io/projected/5c11b2df-1fc1-40ca-86f0-315f19e683e7-kube-api-access-4nq29\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.989368 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.989173 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-dshm\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.989425 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.989376 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-home\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.989506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.989425 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tmp-dir\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.989506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.989498 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-model-cache\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.991464 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.991438 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-dshm\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:16.991765 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:16.991750 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tls-certs\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:17.004821 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:17.004798 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nq29\" (UniqueName: \"kubernetes.io/projected/5c11b2df-1fc1-40ca-86f0-315f19e683e7-kube-api-access-4nq29\") pod \"premium-w0-7bf66c-kserve-78c757bc48-4rfsn\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:17.097675 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:17.097633 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:17.379151 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:17.379063 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn"] Oct 31 13:11:17.381365 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:11:17.381325 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c11b2df_1fc1_40ca_86f0_315f19e683e7.slice/crio-d5463b98a33a3799af07a158a57e50f4a45f231aaff1ede19830e1101f527aef WatchSource:0}: Error finding container d5463b98a33a3799af07a158a57e50f4a45f231aaff1ede19830e1101f527aef: Status 404 returned error can't find the container with id d5463b98a33a3799af07a158a57e50f4a45f231aaff1ede19830e1101f527aef Oct 31 13:11:17.802008 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:17.801920 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" event={"ID":"5c11b2df-1fc1-40ca-86f0-315f19e683e7","Type":"ContainerStarted","Data":"42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2"} Oct 31 13:11:17.802008 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:17.801965 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" event={"ID":"5c11b2df-1fc1-40ca-86f0-315f19e683e7","Type":"ContainerStarted","Data":"d5463b98a33a3799af07a158a57e50f4a45f231aaff1ede19830e1101f527aef"} Oct 31 13:11:17.823429 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:17.823378 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podStartSLOduration=1.823364797 podStartE2EDuration="1.823364797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:17.822771709 +0000 UTC m=+694.667938128" watchObservedRunningTime="2026-10-31 13:11:17.823364797 +0000 UTC m=+694.668531216" Oct 31 13:11:18.098369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:18.098330 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:18.785002 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:18.784968 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:18.798802 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:18.798771 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:18.823030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:18.823002 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:11:23.835419 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:11:23.835384 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:11:24.628535 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.628503 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc"] Oct 31 13:11:24.632490 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.632459 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.640822 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.640797 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2a262a\"/\"kube-root-ca.crt\"" Oct 31 13:11:24.640956 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.640885 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2a262a\"/\"openshift-service-ca.crt\"" Oct 31 13:11:24.641820 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.641779 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2a262a\"/\"default-dockercfg-szbg5\"" Oct 31 13:11:24.642017 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.641966 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2a262a\"/\"premium-w2-2a262a-kserve-self-signed-certs\"" Oct 31 13:11:24.643394 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.643371 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc"] Oct 31 13:11:24.797139 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.797108 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tls-certs\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.797320 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.797238 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tmp-dir\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.797320 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.797301 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-home\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.797508 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.797326 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-model-cache\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.797508 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.797441 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2lg\" (UniqueName: \"kubernetes.io/projected/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-kube-api-access-tk2lg\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.797593 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.797542 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-dshm\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.898847 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.898765 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tls-certs\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.898847 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.898833 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tmp-dir\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.899348 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.898875 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-home\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.899348 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.898896 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-model-cache\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.899348 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.898921 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2lg\" (UniqueName: \"kubernetes.io/projected/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-kube-api-access-tk2lg\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.899348 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.898956 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-dshm\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.899348 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.899302 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-model-cache\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.899582 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.899367 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tmp-dir\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.899582 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.899466 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-home\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.901193 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.901168 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-dshm\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.901547 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.901525 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tls-certs\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.912116 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.912086 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2lg\" (UniqueName: \"kubernetes.io/projected/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-kube-api-access-tk2lg\") pod \"premium-w2-2a262a-kserve-99997b4d-8cpvc\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:24.945641 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:24.945618 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:25.223892 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:25.223866 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc"] Oct 31 13:11:25.225528 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:11:25.225495 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7af893f2_4ce1_4c3b_8d4b_05b17617ab5d.slice/crio-0b785328b817ec5ebb291937082239bfe635fc4461c43c48f99194c2b742c9f2 WatchSource:0}: Error finding container 0b785328b817ec5ebb291937082239bfe635fc4461c43c48f99194c2b742c9f2: Status 404 returned error can't find the container with id 0b785328b817ec5ebb291937082239bfe635fc4461c43c48f99194c2b742c9f2 Oct 31 13:11:25.846426 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:25.846392 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" event={"ID":"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d","Type":"ContainerStarted","Data":"f13423420aec1bf34dd75dd62961e2df3b2567c0b2b297786aa23b43593475fd"} Oct 31 13:11:25.846711 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:25.846433 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" event={"ID":"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d","Type":"ContainerStarted","Data":"0b785328b817ec5ebb291937082239bfe635fc4461c43c48f99194c2b742c9f2"} Oct 31 13:11:25.870094 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:25.870050 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podStartSLOduration=1.870034389 podStartE2EDuration="1.870034389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:25.868580006 +0000 UTC m=+702.713746547" watchObservedRunningTime="2026-10-31 13:11:25.870034389 +0000 UTC m=+702.715200807" Oct 31 13:11:25.945907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:25.945878 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:27.098519 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:27.098460 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:27.111077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:27.111051 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:27.868793 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:27.868767 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:11:31.401579 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:31.401545 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:11:34.946232 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:34.946196 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:34.959176 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:34.959150 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:35.912021 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:35.911993 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:11:53.831845 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:11:53.831806 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:11:55.933437 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:11:55.933402 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:12:06.820920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.820885 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67"] Oct 31 13:12:06.824731 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.824711 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.833557 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.833535 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-79c6ff-openshift-default-dockercfg-tkwch\"" Oct 31 13:12:06.836636 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.836612 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67"] Oct 31 13:12:06.873793 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.873760 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8e330f-01c1-435e-b684-56a1ed9208b7-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.873959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.873812 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.874027 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.873971 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.874027 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.874017 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.874121 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.874037 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.874155 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.874131 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.874192 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.874159 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.874230 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.874198 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kbx\" (UniqueName: \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-kube-api-access-p7kbx\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.874265 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.874231 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.889637 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.889609 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5"] Oct 31 13:12:06.893290 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.893271 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:06.908937 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.908908 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5"] Oct 31 13:12:06.937638 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.937607 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-004f64-openshift-default-dockercfg-k4xnk\"" Oct 31 13:12:06.975886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.975854 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8e330f-01c1-435e-b684-56a1ed9208b7-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976047 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.975898 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48b8206-4423-4c72-8ade-a37075d55f97-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:06.976047 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.975942 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976047 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.975971 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:06.976172 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976129 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48b8206-4423-4c72-8ade-a37075d55f97-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:06.976207 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976196 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:06.976260 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976224 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976334 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976317 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:06.976392 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976354 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:06.976471 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976447 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976571 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976552 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976621 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976573 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976657 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976645 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976691 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976669 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8e330f-01c1-435e-b684-56a1ed9208b7-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976691 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976677 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976763 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976717 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrr8x\" (UniqueName: \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-kube-api-access-zrr8x\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:06.976763 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976753 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976833 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976760 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:06.976873 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976843 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:06.976922 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976903 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kbx\" (UniqueName: \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-kube-api-access-p7kbx\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976964 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976921 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.976964 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.976938 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.977060 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.977038 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.978277 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.978262 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.979015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.978999 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.989932 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.989908 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kbx\" (UniqueName: \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-kube-api-access-p7kbx\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:06.989932 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:06.989926 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:07.077993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.077906 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.077993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.077961 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.077993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.077982 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.078277 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.078034 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrr8x\" (UniqueName: \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-kube-api-access-zrr8x\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.078277 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.078061 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.078277 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.078094 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.078277 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.078141 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48b8206-4423-4c72-8ade-a37075d55f97-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.078277 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.078180 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.078277 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.078216 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48b8206-4423-4c72-8ade-a37075d55f97-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.078564 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.078351 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.078719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.078693 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.078865 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.078754 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.078927 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.078859 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.079049 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.079026 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48b8206-4423-4c72-8ade-a37075d55f97-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.080411 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.080390 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.080852 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.080831 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48b8206-4423-4c72-8ade-a37075d55f97-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.093437 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.093410 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.093886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.093866 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrr8x\" (UniqueName: \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-kube-api-access-zrr8x\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.140147 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.140124 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:07.209157 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.208960 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:07.422882 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.422857 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67"] Oct 31 13:12:07.429904 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.429873 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:07.430017 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.429941 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:07.430017 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.429976 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:07.495774 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.495751 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5"] Oct 31 13:12:07.497000 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:12:07.496972 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48b8206_4423_4c72_8ade_a37075d55f97.slice/crio-a1dae318c2981c386923417a0da6611f491e5ecedb966454563be33b8d270c29 WatchSource:0}: Error finding container a1dae318c2981c386923417a0da6611f491e5ecedb966454563be33b8d270c29: Status 404 returned error can't find the container with id a1dae318c2981c386923417a0da6611f491e5ecedb966454563be33b8d270c29 Oct 31 13:12:07.499649 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.499611 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:07.499720 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.499697 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:07.499758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:07.499738 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:08.095780 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.095736 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" event={"ID":"f48b8206-4423-4c72-8ade-a37075d55f97","Type":"ContainerStarted","Data":"103b758424a7ec7fd142f8bff3a018b48d84fdef3ff7a5af3c11459abd0e860a"} Oct 31 13:12:08.096293 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.095815 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" event={"ID":"f48b8206-4423-4c72-8ade-a37075d55f97","Type":"ContainerStarted","Data":"a1dae318c2981c386923417a0da6611f491e5ecedb966454563be33b8d270c29"} Oct 31 13:12:08.098607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.098560 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" event={"ID":"2f8e330f-01c1-435e-b684-56a1ed9208b7","Type":"ContainerStarted","Data":"a79da1681beb6c1c6783c437edde99b3867dce095d95539245c5accca67aae3a"} Oct 31 13:12:08.098730 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.098616 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" event={"ID":"2f8e330f-01c1-435e-b684-56a1ed9208b7","Type":"ContainerStarted","Data":"1554f90fec441436043b0b2c314b4179e4cc2d3f2e37043399d2d80ad175526a"} Oct 31 13:12:08.124570 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.124514 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" podStartSLOduration=2.124468911 podStartE2EDuration="2.124468911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:08.121307982 +0000 UTC m=+744.966474436" watchObservedRunningTime="2026-10-31 13:12:08.124468911 +0000 UTC m=+744.969635329" Oct 31 13:12:08.181816 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.172820 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:08.183632 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.182468 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 31 13:12:08.183632 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.182551 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 31 13:12:08.189902 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.189846 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" podStartSLOduration=2.189822032 podStartE2EDuration="2.189822032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:08.184158975 +0000 UTC m=+745.029325401" watchObservedRunningTime="2026-10-31 13:12:08.189822032 +0000 UTC m=+745.034988452" Oct 31 13:12:08.209693 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.209621 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:08.210770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.210567 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 13:12:08.210770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:08.210606 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 13:12:09.164501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:09.160497 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 31 13:12:09.164501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:09.160570 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 31 13:12:09.210318 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:09.210285 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 13:12:09.210503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:09.210350 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 13:12:10.143001 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:10.142961 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 31 13:12:10.143293 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:10.143265 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 31 13:12:10.210553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:10.210517 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 13:12:10.210942 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:10.210575 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 13:12:11.155501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:11.147930 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 31 13:12:11.155501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:11.147991 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 31 13:12:11.209448 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:11.209408 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 13:12:11.209671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:11.209463 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 13:12:12.151923 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:12.151885 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 31 13:12:12.152537 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:12.152501 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 31 13:12:12.209914 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:12.209878 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 13:12:12.210103 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:12.209934 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 13:12:13.146870 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:13.146831 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 31 13:12:13.147216 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:13.147173 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 31 13:12:13.209716 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:13.209681 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 13:12:13.210195 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:13.210169 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 13:12:14.222330 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:14.222294 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:14.223530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:14.223374 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:14.224765 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:14.224748 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:12:14.294440 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:14.294403 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:14.295713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:14.295691 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:14.296377 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:14.296353 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:12:23.831152 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:12:23.831114 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:12:36.606649 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.606607 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r"] Oct 31 13:12:36.613093 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.613069 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.737530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.713204 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.737530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.713290 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.737530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.713324 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.737530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.713355 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.737530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.713397 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b015433d-b413-49a1-a515-f903370ba0e0-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.737530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.713517 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.737530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.713569 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.737530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.713626 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n962m\" (UniqueName: \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-kube-api-access-n962m\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.737530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.713666 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b015433d-b413-49a1-a515-f903370ba0e0-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.737530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.735559 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r"] Oct 31 13:12:36.829588 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.829548 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.838864 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.838836 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.846101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.846079 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.846270 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.846252 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.846575 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.846553 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b015433d-b413-49a1-a515-f903370ba0e0-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.847320 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.841079 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.847320 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.839651 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.848004 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.847981 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.848177 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.848159 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.848338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.848321 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n962m\" (UniqueName: \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-kube-api-access-n962m\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.848463 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.848446 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b015433d-b413-49a1-a515-f903370ba0e0-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.849363 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.849340 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b015433d-b413-49a1-a515-f903370ba0e0-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.850973 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.849781 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.851075 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.850017 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.871373 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.867109 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b015433d-b413-49a1-a515-f903370ba0e0-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.898496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.887841 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n962m\" (UniqueName: \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-kube-api-access-n962m\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.898496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.894961 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:36.898496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:36.895158 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:37.154980 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:37.154888 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:37.451872 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:37.451786 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r"] Oct 31 13:12:37.474649 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:12:37.474618 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb015433d_b413_49a1_a515_f903370ba0e0.slice/crio-90986d56ce35d781595089e921e0e96c27367cb86501d7c58c1e276b84e4a8c9 WatchSource:0}: Error finding container 90986d56ce35d781595089e921e0e96c27367cb86501d7c58c1e276b84e4a8c9: Status 404 returned error can't find the container with id 90986d56ce35d781595089e921e0e96c27367cb86501d7c58c1e276b84e4a8c9 Oct 31 13:12:37.477584 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:37.476874 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:37.477584 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:37.476987 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:37.477584 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:37.477034 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:38.314198 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:38.314164 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" event={"ID":"b015433d-b413-49a1-a515-f903370ba0e0","Type":"ContainerStarted","Data":"7e6c4dab7f1c7226a882c66666f84a522da59b8ff7c77fc92cc4386b627b7bf4"} Oct 31 13:12:38.315176 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:38.315156 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" event={"ID":"b015433d-b413-49a1-a515-f903370ba0e0","Type":"ContainerStarted","Data":"90986d56ce35d781595089e921e0e96c27367cb86501d7c58c1e276b84e4a8c9"} Oct 31 13:12:38.351519 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:38.345921 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" podStartSLOduration=2.345904313 podStartE2EDuration="2.345904313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:38.342966263 +0000 UTC m=+775.188132684" watchObservedRunningTime="2026-10-31 13:12:38.345904313 +0000 UTC m=+775.191070732" Oct 31 13:12:39.159501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:39.155546 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:39.160647 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:39.160615 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 13:12:39.160860 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:39.160832 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 13:12:40.160500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:40.157863 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 13:12:40.160500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:40.157931 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 13:12:41.164071 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:41.164028 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 13:12:41.164601 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:41.164104 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 13:12:42.160610 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:42.160570 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 13:12:42.160889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:42.160861 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 13:12:42.587313 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:42.587273 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:12:43.164497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:43.163851 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 13:12:43.164497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:43.163917 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 13:12:44.155498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:44.155437 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 13:12:44.155964 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:44.155529 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 13:12:45.162838 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:45.162807 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:45.163325 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:45.163212 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:45.164769 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:45.164741 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:12:52.166500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.166049 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp"] Oct 31 13:12:52.235501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.234737 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp"] Oct 31 13:12:52.260500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.235682 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.360672 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.336379 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.373915 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.373880 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.380094 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.380070 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.380267 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.380249 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.380455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.380437 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.384816 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.384796 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.385016 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.384996 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.385268 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.385248 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.385841 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.385820 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk4h4\" (UniqueName: \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-kube-api-access-sk4h4\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.496549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.495158 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.496549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.495210 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.496549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.495241 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.496549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.495314 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.496549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.495354 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.496549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.495395 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.496549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.495465 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.496549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.495519 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk4h4\" (UniqueName: \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-kube-api-access-sk4h4\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.496549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.495588 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.496549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.496083 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.497617 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.497589 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.506229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.498294 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.506377 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.498584 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.506463 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.506148 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.530795 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.530763 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.553146 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.553119 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.559506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.559149 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk4h4\" (UniqueName: \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-kube-api-access-sk4h4\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.574189 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.574163 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:52.681813 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:52.681771 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:53.463501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:53.447800 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:12:53.589670 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:12:53.583107 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09eedd25_a0ab_4d2e_85d1_62ee9ad2a846.slice/crio-d0640bf81351515b5d3e1dca25de3a26af0f4238b03582b9eef5ddf848a0726b WatchSource:0}: Error finding container d0640bf81351515b5d3e1dca25de3a26af0f4238b03582b9eef5ddf848a0726b: Status 404 returned error can't find the container with id d0640bf81351515b5d3e1dca25de3a26af0f4238b03582b9eef5ddf848a0726b Oct 31 13:12:53.623898 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:53.623855 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:53.624134 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:53.624106 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:53.624308 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:53.624285 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:53.708720 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:53.708684 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp"] Oct 31 13:12:54.034755 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:12:54.034645 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:12:54.626186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:54.626149 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" event={"ID":"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846","Type":"ContainerStarted","Data":"04ba6391519ea3d7d03cbd11f1e01999258e531c60f8bd84d0de8f4529b5ff03"} Oct 31 13:12:54.626595 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:54.626231 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" event={"ID":"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846","Type":"ContainerStarted","Data":"d0640bf81351515b5d3e1dca25de3a26af0f4238b03582b9eef5ddf848a0726b"} Oct 31 13:12:54.656782 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:54.656727 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" podStartSLOduration=2.656713008 podStartE2EDuration="2.656713008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:54.654183386 +0000 UTC m=+791.499349805" watchObservedRunningTime="2026-10-31 13:12:54.656713008 +0000 UTC m=+791.501879481" Oct 31 13:12:54.683028 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:54.683002 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:12:55.572598 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.572565 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9"] Oct 31 13:12:55.577242 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.577220 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.585129 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.585099 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.585261 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.585132 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.585261 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.585179 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.585347 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.585281 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.585392 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.585350 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.585441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.585416 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.585562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.585535 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbpb\" (UniqueName: \"kubernetes.io/projected/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-kube-api-access-qpbpb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.585660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.585641 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.585698 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.585679 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.591898 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.591871 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9"] Oct 31 13:12:55.610633 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.610600 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv"] Oct 31 13:12:55.615470 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.615446 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.638439 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.638412 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv"] Oct 31 13:12:55.684251 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.684219 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 13:12:55.684436 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.684275 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 13:12:55.686792 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.686764 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.686919 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.686800 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.686919 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.686831 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.687022 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.686930 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.687022 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.686996 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.687104 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687023 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.687104 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687069 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.687217 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687200 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbpb\" (UniqueName: \"kubernetes.io/projected/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-kube-api-access-qpbpb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.687268 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687235 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.687391 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687372 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.687512 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687399 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.687512 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687437 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.687512 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687498 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.687683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687522 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.687683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687545 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.687683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687553 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.687683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687593 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.687683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687623 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5hnq\" (UniqueName: \"kubernetes.io/projected/3defcda2-f67c-4057-b3c4-ec130bbd4e91-kube-api-access-s5hnq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.687930 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687704 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.687930 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.687826 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.688131 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.688104 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.688231 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.688209 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.688417 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.688392 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.689525 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.689469 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.690271 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.690240 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.704100 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.703589 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbpb\" (UniqueName: \"kubernetes.io/projected/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-kube-api-access-qpbpb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.711970 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.711948 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-5v9c9\" (UID: \"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.789333 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789298 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.789529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789349 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.789595 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789524 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.789595 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789579 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.789679 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789597 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.789679 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789640 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.789837 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789813 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.789907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789825 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.789907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789855 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.790018 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789916 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.790018 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789956 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5hnq\" (UniqueName: \"kubernetes.io/projected/3defcda2-f67c-4057-b3c4-ec130bbd4e91-kube-api-access-s5hnq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.790018 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.789998 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.790174 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.790161 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.790282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.790262 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.791992 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.791970 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.792089 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.791994 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.802679 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.802643 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3defcda2-f67c-4057-b3c4-ec130bbd4e91-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.805229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.805206 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5hnq\" (UniqueName: \"kubernetes.io/projected/3defcda2-f67c-4057-b3c4-ec130bbd4e91-kube-api-access-s5hnq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xvwjv\" (UID: \"3defcda2-f67c-4057-b3c4-ec130bbd4e91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:55.898534 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.898415 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:55.930509 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:55.930462 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:56.236573 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.236548 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9"] Oct 31 13:12:56.241507 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:12:56.238570 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade2f44e_b0bb_4004_a5c6_3d84e9e80fcc.slice/crio-c19609219728b53f1567954a19191925a49ff417660dcaa406d587ee86ae8081 WatchSource:0}: Error finding container c19609219728b53f1567954a19191925a49ff417660dcaa406d587ee86ae8081: Status 404 returned error can't find the container with id c19609219728b53f1567954a19191925a49ff417660dcaa406d587ee86ae8081 Oct 31 13:12:56.263719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.252865 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:56.263719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.252956 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:56.263719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.253003 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:56.267560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.267537 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv"] Oct 31 13:12:56.268781 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:12:56.268760 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3defcda2_f67c_4057_b3c4_ec130bbd4e91.slice/crio-c9d39ada41426d8b9ff2814414fede08c5828b2888d8519a72b71734447ad931 WatchSource:0}: Error finding container c9d39ada41426d8b9ff2814414fede08c5828b2888d8519a72b71734447ad931: Status 404 returned error can't find the container with id c9d39ada41426d8b9ff2814414fede08c5828b2888d8519a72b71734447ad931 Oct 31 13:12:56.272723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.272696 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:56.272791 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.272755 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:56.272832 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.272788 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:12:56.653861 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.653823 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" event={"ID":"3defcda2-f67c-4057-b3c4-ec130bbd4e91","Type":"ContainerStarted","Data":"445e2ae617f7bddd7af7b188f978c95293382608ab5c14691e1802a52b83eab8"} Oct 31 13:12:56.663713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.663680 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" event={"ID":"3defcda2-f67c-4057-b3c4-ec130bbd4e91","Type":"ContainerStarted","Data":"c9d39ada41426d8b9ff2814414fede08c5828b2888d8519a72b71734447ad931"} Oct 31 13:12:56.667496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.666091 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" event={"ID":"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc","Type":"ContainerStarted","Data":"986f9448c77b3f68d64c14ca4440f776c1200bc6f721b30600731914731b2d13"} Oct 31 13:12:56.667496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.666139 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" event={"ID":"ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc","Type":"ContainerStarted","Data":"c19609219728b53f1567954a19191925a49ff417660dcaa406d587ee86ae8081"} Oct 31 13:12:56.683050 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.683010 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 13:12:56.683217 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.683061 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 13:12:56.723754 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.723695 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" podStartSLOduration=1.7236812289999999 podStartE2EDuration="1.723681229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:56.721522585 +0000 UTC m=+793.566689005" watchObservedRunningTime="2026-10-31 13:12:56.723681229 +0000 UTC m=+793.568847646" Oct 31 13:12:56.725149 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.725110 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" podStartSLOduration=1.725097412 podStartE2EDuration="1.725097412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:56.696140597 +0000 UTC m=+793.541307014" watchObservedRunningTime="2026-10-31 13:12:56.725097412 +0000 UTC m=+793.570263831" Oct 31 13:12:56.899391 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.899355 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:12:56.931160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:56.931052 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:12:57.686378 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:57.685909 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 13:12:57.686378 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:57.685971 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 13:12:57.902635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:57.902598 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5v9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 13:12:57.902914 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:57.902884 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" podUID="ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 13:12:57.931943 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:57.931904 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xvwjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 13:12:57.932117 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:57.931970 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" podUID="3defcda2-f67c-4057-b3c4-ec130bbd4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 13:12:58.684895 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:58.684855 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 13:12:58.685218 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:58.685171 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 13:12:58.900928 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:58.900888 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5v9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 13:12:58.902264 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:58.902231 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" podUID="ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 13:12:58.930763 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:58.930733 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xvwjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 13:12:58.930906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:58.930777 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" podUID="3defcda2-f67c-4057-b3c4-ec130bbd4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 13:12:59.685500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:59.684896 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 13:12:59.685500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:59.684959 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 13:12:59.922203 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:59.922149 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5v9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 13:12:59.922733 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:59.922242 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" podUID="ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 13:12:59.931507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:59.931053 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xvwjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 13:12:59.931507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:12:59.931135 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" podUID="3defcda2-f67c-4057-b3c4-ec130bbd4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 13:13:00.695649 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:00.695611 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 13:13:00.695981 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:00.695939 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 13:13:00.899096 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:00.899051 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5v9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 13:13:00.899307 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:00.899121 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" podUID="ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 13:13:00.931487 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:00.931449 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xvwjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 13:13:00.931954 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:00.931537 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" podUID="3defcda2-f67c-4057-b3c4-ec130bbd4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 13:13:01.708949 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:01.708920 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:13:01.709424 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:01.709374 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:13:01.710208 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:01.710186 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:13:01.899134 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:01.899100 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5v9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 13:13:01.899298 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:01.899178 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" podUID="ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 13:13:01.931465 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:01.931436 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xvwjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 13:13:01.931680 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:01.931658 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" podUID="3defcda2-f67c-4057-b3c4-ec130bbd4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 13:13:02.899020 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:02.898977 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5v9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 13:13:02.899312 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:02.899056 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" podUID="ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 13:13:02.931686 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:02.931637 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xvwjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 13:13:02.932186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:02.931697 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" podUID="3defcda2-f67c-4057-b3c4-ec130bbd4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 13:13:03.899439 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:03.899399 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5v9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 13:13:03.899641 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:03.899486 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" podUID="ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 13:13:03.931797 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:03.931758 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xvwjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 13:13:03.932195 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:03.931833 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" podUID="3defcda2-f67c-4057-b3c4-ec130bbd4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 13:13:04.648438 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.648397 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw"] Oct 31 13:13:04.653342 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.653309 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.663221 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.663195 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2a262a\"/\"distinct-w2-2a262a-kserve-self-signed-certs\"" Oct 31 13:13:04.669683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.669652 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw"] Oct 31 13:13:04.740157 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.740127 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fec81e-a3da-4857-9aa8-555bd89744fc-tls-certs\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.740338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.740269 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-tmp-dir\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.740488 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.740443 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-model-cache\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.740567 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.740503 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-home\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.740659 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.740633 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqmhj\" (UniqueName: \"kubernetes.io/projected/78fec81e-a3da-4857-9aa8-555bd89744fc-kube-api-access-wqmhj\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.740727 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.740683 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-dshm\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.845270 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.845225 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqmhj\" (UniqueName: \"kubernetes.io/projected/78fec81e-a3da-4857-9aa8-555bd89744fc-kube-api-access-wqmhj\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.845713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.845686 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-dshm\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.845938 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.845924 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fec81e-a3da-4857-9aa8-555bd89744fc-tls-certs\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.846105 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.846088 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-tmp-dir\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.846320 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.846301 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-model-cache\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.846431 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.846417 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-home\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.854985 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.854907 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-tmp-dir\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.856254 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.856190 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fec81e-a3da-4857-9aa8-555bd89744fc-tls-certs\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.887517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.859904 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-model-cache\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.887517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.860118 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-home\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.887517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.864034 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqmhj\" (UniqueName: \"kubernetes.io/projected/78fec81e-a3da-4857-9aa8-555bd89744fc-kube-api-access-wqmhj\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.888548 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.888461 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-dshm\") pod \"distinct-w2-2a262a-kserve-6f757595d7-x5jpw\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:04.981825 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:04.981732 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:05.304879 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.302921 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw"] Oct 31 13:13:05.321165 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:05.320961 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78fec81e_a3da_4857_9aa8_555bd89744fc.slice/crio-bbb4b0e2eccf70a99a271a02031f6fec18366b0716dd5e45730a3b2b4020c4f8 WatchSource:0}: Error finding container bbb4b0e2eccf70a99a271a02031f6fec18366b0716dd5e45730a3b2b4020c4f8: Status 404 returned error can't find the container with id bbb4b0e2eccf70a99a271a02031f6fec18366b0716dd5e45730a3b2b4020c4f8 Oct 31 13:13:05.360664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.359409 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:13:05.360664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.360120 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:13:05.361055 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.361035 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" Oct 31 13:13:05.490642 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.490611 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:13:05.491757 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.491736 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:13:05.493981 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.493930 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" Oct 31 13:13:05.650095 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.650065 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm"] Oct 31 13:13:05.653813 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.653793 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.658626 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.658605 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7bf66c\"/\"distinct-w0-7bf66c-kserve-self-signed-certs\"" Oct 31 13:13:05.668261 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.668239 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm"] Oct 31 13:13:05.770932 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.770902 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tls-certs\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.771554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.771007 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-model-cache\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.771554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.771052 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-home\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.771554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.771097 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrr6\" (UniqueName: \"kubernetes.io/projected/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-kube-api-access-lhrr6\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.771554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.771136 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-dshm\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.771554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.771214 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tmp-dir\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.831070 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.831035 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" event={"ID":"78fec81e-a3da-4857-9aa8-555bd89744fc","Type":"ContainerStarted","Data":"9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650"} Oct 31 13:13:05.831070 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.831076 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" event={"ID":"78fec81e-a3da-4857-9aa8-555bd89744fc","Type":"ContainerStarted","Data":"bbb4b0e2eccf70a99a271a02031f6fec18366b0716dd5e45730a3b2b4020c4f8"} Oct 31 13:13:05.853631 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.853577 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podStartSLOduration=1.8535635259999999 podStartE2EDuration="1.853563526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:05.850850213 +0000 UTC m=+802.696016632" watchObservedRunningTime="2026-10-31 13:13:05.853563526 +0000 UTC m=+802.698729963" Oct 31 13:13:05.871958 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.871930 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tls-certs\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.872095 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.872011 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-model-cache\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.872095 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.872036 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-home\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.872095 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.872067 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrr6\" (UniqueName: \"kubernetes.io/projected/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-kube-api-access-lhrr6\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.872200 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.872182 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-dshm\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.872275 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.872260 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tmp-dir\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.872517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.872491 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-home\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.872643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.872534 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-model-cache\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.872770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.872750 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tmp-dir\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.874388 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.874365 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-dshm\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.874605 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.874589 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tls-certs\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.885969 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.885946 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrr6\" (UniqueName: \"kubernetes.io/projected/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-kube-api-access-lhrr6\") pod \"distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.966039 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.965959 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:05.982898 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:05.982822 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:06.262902 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:06.262875 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm"] Oct 31 13:13:06.263882 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:06.263855 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0d5e01c_7a06_4122_b5d9_0e76ff7b0de2.slice/crio-264827dad23a248356059e56df9968a2c90eb0cc35e6bde27953f46a04260e91 WatchSource:0}: Error finding container 264827dad23a248356059e56df9968a2c90eb0cc35e6bde27953f46a04260e91: Status 404 returned error can't find the container with id 264827dad23a248356059e56df9968a2c90eb0cc35e6bde27953f46a04260e91 Oct 31 13:13:06.847415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:06.847364 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" event={"ID":"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2","Type":"ContainerStarted","Data":"5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a"} Oct 31 13:13:06.847636 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:06.847428 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" event={"ID":"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2","Type":"ContainerStarted","Data":"264827dad23a248356059e56df9968a2c90eb0cc35e6bde27953f46a04260e91"} Oct 31 13:13:06.875341 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:06.875269 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podStartSLOduration=1.875255984 podStartE2EDuration="1.875255984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:06.872953277 +0000 UTC m=+803.718119694" watchObservedRunningTime="2026-10-31 13:13:06.875255984 +0000 UTC m=+803.720422404" Oct 31 13:13:06.966340 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:06.966291 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:06.982196 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:06.982157 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn"] Oct 31 13:13:06.987515 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:06.987492 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.065174 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.065140 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6"] Oct 31 13:13:07.075778 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.075742 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6"] Oct 31 13:13:07.076001 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.075975 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.095615 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.095576 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.095781 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.095659 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.095781 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.095691 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.095781 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.095761 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.095995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.095807 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddbe95-4c3e-494d-9257-accc94089b63-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.095995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.095841 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddbe95-4c3e-494d-9257-accc94089b63-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.095995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.095883 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.095995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.095928 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.096211 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.096013 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvc6\" (UniqueName: \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-kube-api-access-cpvc6\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.101597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.101556 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn"] Oct 31 13:13:07.197702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.197667 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.197893 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.197716 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.197893 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.197743 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.197893 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.197761 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.197893 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.197777 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.198123 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.197898 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4fjl\" (UniqueName: \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-kube-api-access-b4fjl\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.198123 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.197937 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.198123 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.197962 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.198123 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198099 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddbe95-4c3e-494d-9257-accc94089b63-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.198123 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198123 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.198336 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198138 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.198336 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198157 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.198336 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198209 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.198336 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198221 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddbe95-4c3e-494d-9257-accc94089b63-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.198336 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198254 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.198336 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198277 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.198336 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198303 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.198602 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198348 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.198602 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198407 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.198602 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198438 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.198602 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198581 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvc6\" (UniqueName: \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-kube-api-access-cpvc6\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.198788 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198775 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.198842 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.198798 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddbe95-4c3e-494d-9257-accc94089b63-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.200487 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.200453 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.200786 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.200769 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddbe95-4c3e-494d-9257-accc94089b63-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.210700 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.210675 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.211306 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.211287 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvc6\" (UniqueName: \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-kube-api-access-cpvc6\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.299425 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299395 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.299616 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299446 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.299616 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299464 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.299616 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299506 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.299616 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299537 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.299616 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299580 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.299857 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299720 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.299857 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299755 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.299857 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299787 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fjl\" (UniqueName: \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-kube-api-access-b4fjl\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.300010 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299914 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.300010 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.299967 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.300157 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.300124 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.300239 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.300218 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.300335 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.300315 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.301496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.301461 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:07.301861 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.301837 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.302280 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.302261 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.313173 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.313153 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fjl\" (UniqueName: \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-kube-api-access-b4fjl\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.313340 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.313324 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.409836 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.409789 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:07.586241 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.586216 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn"] Oct 31 13:13:07.589151 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:07.589113 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ddbe95_4c3e_494d_9257_accc94089b63.slice/crio-4a2b90fb4c948b1cad1bcd7d48721dce2a5b69f276c6f5334093ea5f46efd60f WatchSource:0}: Error finding container 4a2b90fb4c948b1cad1bcd7d48721dce2a5b69f276c6f5334093ea5f46efd60f: Status 404 returned error can't find the container with id 4a2b90fb4c948b1cad1bcd7d48721dce2a5b69f276c6f5334093ea5f46efd60f Oct 31 13:13:07.591855 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.591818 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:07.591934 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.591886 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:07.591934 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.591916 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:07.722564 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.711266 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6"] Oct 31 13:13:07.726606 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:07.726568 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b70fcb_394c_4c0d_baa8_d0c8323c0723.slice/crio-300c4d51fcedd828742c7bc5711acf40606a3b0b5f78de502c35dbd769806d3a WatchSource:0}: Error finding container 300c4d51fcedd828742c7bc5711acf40606a3b0b5f78de502c35dbd769806d3a: Status 404 returned error can't find the container with id 300c4d51fcedd828742c7bc5711acf40606a3b0b5f78de502c35dbd769806d3a Oct 31 13:13:07.730000 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.729961 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:07.730088 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.730037 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:07.730088 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.730071 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:07.859095 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.858159 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" event={"ID":"32ddbe95-4c3e-494d-9257-accc94089b63","Type":"ContainerStarted","Data":"b2016364e14ce472d4cc9c0007791ccaf0bb3a4a81338886324bb6416fea4620"} Oct 31 13:13:07.859095 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.858261 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" event={"ID":"32ddbe95-4c3e-494d-9257-accc94089b63","Type":"ContainerStarted","Data":"4a2b90fb4c948b1cad1bcd7d48721dce2a5b69f276c6f5334093ea5f46efd60f"} Oct 31 13:13:07.861560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.861527 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" event={"ID":"95b70fcb-394c-4c0d-baa8-d0c8323c0723","Type":"ContainerStarted","Data":"97ede513ae6cfd2cae07d632c12691bd4b52079aae5e43ca4ac55257408be486"} Oct 31 13:13:07.861682 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.861573 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" event={"ID":"95b70fcb-394c-4c0d-baa8-d0c8323c0723","Type":"ContainerStarted","Data":"300c4d51fcedd828742c7bc5711acf40606a3b0b5f78de502c35dbd769806d3a"} Oct 31 13:13:07.901051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.900998 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" podStartSLOduration=1.900983077 podStartE2EDuration="1.900983077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:07.899299729 +0000 UTC m=+804.744466150" watchObservedRunningTime="2026-10-31 13:13:07.900983077 +0000 UTC m=+804.746149495" Oct 31 13:13:07.929141 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:07.929092 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podStartSLOduration=0.929077518 podStartE2EDuration="929.077518ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:07.92526691 +0000 UTC m=+804.770433330" watchObservedRunningTime="2026-10-31 13:13:07.929077518 +0000 UTC m=+804.774243937" Oct 31 13:13:08.302539 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:08.302441 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:08.304581 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:08.304553 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 13:13:08.304719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:08.304604 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 13:13:08.359498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:08.358420 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:13:08.413501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:08.411676 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:08.414098 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:08.414069 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 13:13:08.414340 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:08.414297 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 13:13:09.302614 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:09.302578 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 13:13:09.303142 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:09.302665 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 13:13:09.462662 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:09.462611 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 13:13:09.462928 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:09.462904 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 13:13:10.303500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:10.302201 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 13:13:10.303500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:10.302265 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 13:13:10.413501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:10.412880 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 13:13:10.413501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:10.412958 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 13:13:10.965519 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:10.961775 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 13:13:10.965519 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:10.964730 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 13:13:11.323830 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:11.323782 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 13:13:11.324470 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:11.324416 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 13:13:11.417501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:11.414446 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 13:13:11.417501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:11.414528 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 13:13:11.434500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:11.430594 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:13:12.302161 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:12.302128 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 13:13:12.302335 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:12.302181 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 13:13:12.410608 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:12.410565 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 13:13:12.410991 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:12.410625 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 13:13:13.309102 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:13.306710 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 13:13:13.309102 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:13.306793 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 13:13:13.410750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:13.410716 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 13:13:13.411149 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:13.410782 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 13:13:14.302275 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:14.302240 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 13:13:14.302463 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:14.302299 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 13:13:14.423668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:14.423629 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 13:13:14.424263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:14.424230 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 13:13:14.982015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:14.981977 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:14.994802 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:14.994775 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:15.157622 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:15.157590 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:13:15.410243 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:15.410208 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 13:13:15.410424 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:15.410269 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 13:13:15.966185 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:15.966149 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:15.985659 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:15.985625 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:16.165425 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:16.165397 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:13:16.261671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:16.261593 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:16.261977 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:16.261959 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:16.262634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:16.262617 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:13:16.415074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:16.415044 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:16.415348 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:16.415327 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:16.415963 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:16.415946 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:13:24.073909 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:13:24.073867 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:13:24.167639 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:24.167595 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:13:24.426668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:24.426555 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:13:24.766583 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:24.766459 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:13:24.814832 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:24.814776 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:13:24.863534 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:24.862574 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:13:24.911815 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:24.910569 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 13:13:25.146907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:25.146572 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:13:36.487611 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.487577 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn"] Oct 31 13:13:36.491398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.491376 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.505087 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.505063 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn"] Oct 31 13:13:36.534453 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.534418 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-dshm\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.534652 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.534549 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tls-certs\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.534652 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.534609 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pn6f\" (UniqueName: \"kubernetes.io/projected/0d065383-afaa-4679-9d3e-3da3ad6ddba6-kube-api-access-8pn6f\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.534802 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.534733 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-home\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.534802 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.534789 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tmp-dir\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.534910 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.534849 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-model-cache\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.539154 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.539133 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts"] Oct 31 13:13:36.542798 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.542776 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.545681 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.545661 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2a262a\"/\"distinct-2-w2-2a262a-kserve-self-signed-certs\"" Oct 31 13:13:36.565888 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.565860 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts"] Oct 31 13:13:36.635905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.635864 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.635905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.635906 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tls-certs\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.636159 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.635984 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pn6f\" (UniqueName: \"kubernetes.io/projected/0d065383-afaa-4679-9d3e-3da3ad6ddba6-kube-api-access-8pn6f\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.636159 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636100 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.636159 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636142 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-home\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.636375 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636166 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.636375 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636289 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.636375 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636342 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.636561 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636383 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tmp-dir\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.636561 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636549 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-home\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.636679 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636640 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5jdj\" (UniqueName: \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-kube-api-access-r5jdj\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.636735 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636678 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-model-cache\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.636735 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636679 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tmp-dir\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.636838 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636774 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-dshm\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.636838 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636792 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.636838 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636831 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.636993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636848 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.636993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.636948 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-model-cache\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.638911 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.638886 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tls-certs\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.639105 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.639086 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-dshm\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.663415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.663350 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pn6f\" (UniqueName: \"kubernetes.io/projected/0d065383-afaa-4679-9d3e-3da3ad6ddba6-kube-api-access-8pn6f\") pod \"distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.736403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.735766 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln"] Oct 31 13:13:36.739427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.738426 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.739427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.738495 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.739427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.738546 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.739427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.738655 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.739427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.738696 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.739427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.738724 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.739427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.738795 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.739427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.738868 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jdj\" (UniqueName: \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-kube-api-access-r5jdj\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.739427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.739001 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.740047 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.739900 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.741214 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.741178 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.741573 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.741553 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.741766 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.741283 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.742533 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.738960 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.747074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.747046 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.747074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.747070 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.747973 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.747954 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.771375 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.771341 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln"] Oct 31 13:13:36.783333 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.783295 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.783551 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.783382 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5jdj\" (UniqueName: \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-kube-api-access-r5jdj\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.811032 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.810776 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:36.843780 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.843743 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.843938 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.843899 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.844003 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.843947 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.844089 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.844063 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzjbm\" (UniqueName: \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-kube-api-access-qzjbm\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.844264 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.844122 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.844264 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.844211 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.844363 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.844261 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.844363 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.844284 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.844363 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.844319 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.856321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.856292 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.945628 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.945767 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.945822 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.945859 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjbm\" (UniqueName: \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-kube-api-access-qzjbm\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.945900 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.945958 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.946012 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.946041 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.946099 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.946948 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.947869 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.948557 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.948831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.948633 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.949627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.948919 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.950532 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.950510 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.952331 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.952303 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.960383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.960349 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:36.961515 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:36.961494 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjbm\" (UniqueName: \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-kube-api-access-qzjbm\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:37.096415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.096391 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:37.103172 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.103147 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn"] Oct 31 13:13:37.107211 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:37.107183 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d065383_afaa_4679_9d3e_3da3ad6ddba6.slice/crio-669088fc93d783eaacc16ae4927d339c7641f9e18e00be86d295117d4176632b WatchSource:0}: Error finding container 669088fc93d783eaacc16ae4927d339c7641f9e18e00be86d295117d4176632b: Status 404 returned error can't find the container with id 669088fc93d783eaacc16ae4927d339c7641f9e18e00be86d295117d4176632b Oct 31 13:13:37.164506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.164407 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts"] Oct 31 13:13:37.176860 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:37.173010 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb27e2_2458_4c4f_8b00_b145c9bd8116.slice/crio-f4583b171fb169899d987adcd50a46ef7e143c835a687fd1a513e4c2759d8a4a WatchSource:0}: Error finding container f4583b171fb169899d987adcd50a46ef7e143c835a687fd1a513e4c2759d8a4a: Status 404 returned error can't find the container with id f4583b171fb169899d987adcd50a46ef7e143c835a687fd1a513e4c2759d8a4a Oct 31 13:13:37.177491 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.177439 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:37.177605 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.177543 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:37.177605 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.177586 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:37.344193 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.344151 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" event={"ID":"0d065383-afaa-4679-9d3e-3da3ad6ddba6","Type":"ContainerStarted","Data":"42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a"} Oct 31 13:13:37.344343 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.344200 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" event={"ID":"0d065383-afaa-4679-9d3e-3da3ad6ddba6","Type":"ContainerStarted","Data":"669088fc93d783eaacc16ae4927d339c7641f9e18e00be86d295117d4176632b"} Oct 31 13:13:37.345962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.345935 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" event={"ID":"b1cb27e2-2458-4c4f-8b00-b145c9bd8116","Type":"ContainerStarted","Data":"d7ad38a356eb899a6d1058d464691a536736f97180208dcbce66b71c6079e923"} Oct 31 13:13:37.346100 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.345981 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" event={"ID":"b1cb27e2-2458-4c4f-8b00-b145c9bd8116","Type":"ContainerStarted","Data":"f4583b171fb169899d987adcd50a46ef7e143c835a687fd1a513e4c2759d8a4a"} Oct 31 13:13:37.369633 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.369542 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podStartSLOduration=1.369529254 podStartE2EDuration="1.369529254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:37.366996424 +0000 UTC m=+834.212162841" watchObservedRunningTime="2026-10-31 13:13:37.369529254 +0000 UTC m=+834.214695672" Oct 31 13:13:37.388280 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.388259 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln"] Oct 31 13:13:37.390293 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:37.390268 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c6d6ca_4bc9_4b35_8133_fa2f7637ed40.slice/crio-7fa68ceaf2facb0b02ccf7682b988095272968b587b5930d213724aa2710c5cd WatchSource:0}: Error finding container 7fa68ceaf2facb0b02ccf7682b988095272968b587b5930d213724aa2710c5cd: Status 404 returned error can't find the container with id 7fa68ceaf2facb0b02ccf7682b988095272968b587b5930d213724aa2710c5cd Oct 31 13:13:37.392742 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.392713 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:37.392826 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.392768 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:37.392826 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.392809 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:37.394662 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.394619 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" podStartSLOduration=1.394607492 podStartE2EDuration="1.394607492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:37.393566595 +0000 UTC m=+834.238733015" watchObservedRunningTime="2026-10-31 13:13:37.394607492 +0000 UTC m=+834.239773910" Oct 31 13:13:37.811567 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.811486 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:37.858613 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:37.858574 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:38.401129 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:38.401089 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" event={"ID":"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40","Type":"ContainerStarted","Data":"2b531afbb82da126495810f2dc90b77be0c542ea789237ad4b4b11d3943eda13"} Oct 31 13:13:38.401452 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:38.401426 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" event={"ID":"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40","Type":"ContainerStarted","Data":"7fa68ceaf2facb0b02ccf7682b988095272968b587b5930d213724aa2710c5cd"} Oct 31 13:13:38.430523 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:38.430453 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" podStartSLOduration=2.430440137 podStartE2EDuration="2.430440137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:38.427204769 +0000 UTC m=+835.272371188" watchObservedRunningTime="2026-10-31 13:13:38.430440137 +0000 UTC m=+835.275606612" Oct 31 13:13:38.862612 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:38.862573 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:13:38.863941 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:38.863908 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 13:13:39.097345 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.097311 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:39.098741 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.098717 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 13:13:39.098875 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.098756 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 13:13:39.501533 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.498185 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk"] Oct 31 13:13:39.504785 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.504757 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk"] Oct 31 13:13:39.505102 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.505081 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.514132 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.514052 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7bf66c\"/\"distinct-2-w0-7bf66c-kserve-self-signed-certs\"" Oct 31 13:13:39.616062 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.616017 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-dshm\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.616314 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.616294 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-model-cache\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.616433 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.616420 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tls-certs\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.616646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.616627 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2h4\" (UniqueName: \"kubernetes.io/projected/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-kube-api-access-kf2h4\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.617001 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.616964 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tmp-dir\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.617123 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.617101 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-home\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.719027 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.718986 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tmp-dir\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.719227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.719045 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-home\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.719227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.719119 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-dshm\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.719227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.719146 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-model-cache\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.719227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.719173 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tls-certs\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.719447 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.719285 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2h4\" (UniqueName: \"kubernetes.io/projected/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-kube-api-access-kf2h4\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.719587 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.719560 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-home\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.719940 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.719918 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-model-cache\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.720030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.719985 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tmp-dir\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.724642 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.724617 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tls-certs\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.727436 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.727418 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-dshm\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.733539 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.733517 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2h4\" (UniqueName: \"kubernetes.io/projected/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-kube-api-access-kf2h4\") pod \"distinct-2-w0-7bf66c-kserve-788b84755b-6spwk\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.825847 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.825819 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:39.872613 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.872582 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:13:39.894233 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:39.894188 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 13:13:40.096969 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:40.096882 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 13:13:40.096969 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:40.096952 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 13:13:40.194068 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:40.194015 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk"] Oct 31 13:13:40.195920 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:40.195839 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe6be33c_8d58_4ebf_9f5c_e1e4a6b2b937.slice/crio-08d1787182dbad2c92dc508f3d536e1d88bbac4f1969d5f83587386da33644b3 WatchSource:0}: Error finding container 08d1787182dbad2c92dc508f3d536e1d88bbac4f1969d5f83587386da33644b3: Status 404 returned error can't find the container with id 08d1787182dbad2c92dc508f3d536e1d88bbac4f1969d5f83587386da33644b3 Oct 31 13:13:40.419142 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:40.419068 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" event={"ID":"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937","Type":"ContainerStarted","Data":"2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40"} Oct 31 13:13:40.419142 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:40.419111 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" event={"ID":"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937","Type":"ContainerStarted","Data":"08d1787182dbad2c92dc508f3d536e1d88bbac4f1969d5f83587386da33644b3"} Oct 31 13:13:40.826575 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:40.826541 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:40.860235 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:40.860201 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:13:40.860521 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:40.860490 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 13:13:41.097540 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:41.097434 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 13:13:41.097540 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:41.097519 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 13:13:41.864500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:41.863860 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:13:41.864500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:41.863926 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 13:13:42.097034 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:42.097000 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 13:13:42.097209 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:42.097059 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 13:13:42.856987 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:42.856948 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:13:42.858325 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:42.858288 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 13:13:43.097257 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:43.097223 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 13:13:43.097537 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:43.097512 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 13:13:43.857544 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:43.857460 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:13:43.858065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:43.857562 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 13:13:44.269627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.269535 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:44.270363 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.270335 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:44.271423 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.271401 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:13:44.298846 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.298791 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podStartSLOduration=5.298773301 podStartE2EDuration="5.298773301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:40.476944445 +0000 UTC m=+837.322110864" watchObservedRunningTime="2026-10-31 13:13:44.298773301 +0000 UTC m=+841.143939722" Oct 31 13:13:44.568071 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.568039 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg"] Oct 31 13:13:44.573121 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.573101 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.580060 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.580037 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-426787\"/\"openshift-service-ca.crt\"" Oct 31 13:13:44.580182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.580116 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-426787\"/\"kube-root-ca.crt\"" Oct 31 13:13:44.580182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.580131 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-426787\"/\"default-dockercfg-s79md\"" Oct 31 13:13:44.580182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.580136 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-426787\"/\"distinct-w4-426787-kserve-self-signed-certs\"" Oct 31 13:13:44.584878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.584848 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg"] Oct 31 13:13:44.736792 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.736752 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-model-cache\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.736992 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.736937 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tls-certs\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.737086 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.737067 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-home\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.737142 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.737111 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw5fn\" (UniqueName: \"kubernetes.io/projected/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-kube-api-access-cw5fn\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.737244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.737154 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-dshm\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.737244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.737223 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tmp-dir\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.838244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.838155 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-home\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.838244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.838198 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw5fn\" (UniqueName: \"kubernetes.io/projected/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-kube-api-access-cw5fn\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.838244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.838225 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-dshm\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.838542 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.838269 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tmp-dir\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.838542 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.838325 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-model-cache\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.838542 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.838407 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tls-certs\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.838754 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.838728 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tmp-dir\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.838858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.838829 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-home\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.839008 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.838889 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-model-cache\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.840777 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.840752 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-dshm\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.841247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.841227 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tls-certs\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.851205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.851178 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw5fn\" (UniqueName: \"kubernetes.io/projected/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-kube-api-access-cw5fn\") pod \"distinct-w4-426787-kserve-5d5c54f69-7xpkg\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:44.868342 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.868315 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:44.869958 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.869660 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:44.871063 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.871046 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:13:44.901947 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:44.901924 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:45.210997 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:45.210968 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg"] Oct 31 13:13:45.212443 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:45.212412 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f3ae33d_ed8c_4ce2_ba24_53be34159ba1.slice/crio-fbfd139c64a3f12cd07a56c622386f78c5694039ee93cf1087f76b7f3ba62d9f WatchSource:0}: Error finding container fbfd139c64a3f12cd07a56c622386f78c5694039ee93cf1087f76b7f3ba62d9f: Status 404 returned error can't find the container with id fbfd139c64a3f12cd07a56c622386f78c5694039ee93cf1087f76b7f3ba62d9f Oct 31 13:13:45.496994 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:45.496884 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" event={"ID":"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1","Type":"ContainerStarted","Data":"0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0"} Oct 31 13:13:45.496994 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:45.496954 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" event={"ID":"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1","Type":"ContainerStarted","Data":"fbfd139c64a3f12cd07a56c622386f78c5694039ee93cf1087f76b7f3ba62d9f"} Oct 31 13:13:45.520186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:45.520137 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podStartSLOduration=1.520119556 podStartE2EDuration="1.520119556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:45.519376873 +0000 UTC m=+842.364543293" watchObservedRunningTime="2026-10-31 13:13:45.520119556 +0000 UTC m=+842.365285975" Oct 31 13:13:45.868236 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:45.868206 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:13:45.902417 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:45.902233 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:46.823119 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:46.812496 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:47.014497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:47.013933 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:47.799496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:47.798780 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:13:49.826947 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:49.826915 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:49.839183 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:49.839158 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:50.593489 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:50.593453 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:13:51.827410 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.827380 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht"] Oct 31 13:13:51.831644 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.831624 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:51.854514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.853021 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht"] Oct 31 13:13:51.948269 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.948241 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:51.948383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.948360 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:51.948428 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.948389 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr6nf\" (UniqueName: \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-kube-api-access-pr6nf\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:51.948626 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.948600 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:51.948721 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.948704 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:51.948757 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.948730 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d057adb-f72c-4327-b979-aa61fdf98068-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:51.948757 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.948747 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d057adb-f72c-4327-b979-aa61fdf98068-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:51.948900 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.948767 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:51.948936 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:51.948902 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.050587 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.050550 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.050835 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.050604 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr6nf\" (UniqueName: \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-kube-api-access-pr6nf\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.050835 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.050692 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.050835 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.050764 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.050835 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.050788 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d057adb-f72c-4327-b979-aa61fdf98068-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.050835 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.050813 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d057adb-f72c-4327-b979-aa61fdf98068-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.052167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.050838 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.052167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.050879 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.052167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.050977 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.052167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.051346 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.052167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.051861 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.052167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.051957 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.052167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.052100 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d057adb-f72c-4327-b979-aa61fdf98068-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.052693 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.052668 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.053324 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.053299 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.054995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.054974 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d057adb-f72c-4327-b979-aa61fdf98068-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.063687 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.063666 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.063772 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.063729 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr6nf\" (UniqueName: \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-kube-api-access-pr6nf\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.131708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.131630 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb"] Oct 31 13:13:52.136607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.136579 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.157708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.153876 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb"] Oct 31 13:13:52.176032 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.176002 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:52.259077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.254330 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.259077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.254383 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.259077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.254553 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.259077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.254622 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.259077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.254647 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.259077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.254691 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.259077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.254770 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.259077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.254797 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.259077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.254839 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc7ns\" (UniqueName: \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-kube-api-access-sc7ns\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.259077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.256169 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv"] Oct 31 13:13:52.262847 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.262822 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.271646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.271627 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv"] Oct 31 13:13:52.356229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356192 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.356389 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356330 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.356555 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356535 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.356603 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356571 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.356642 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356599 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.356642 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356625 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.356734 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356651 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.356734 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356693 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.356845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356766 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.356845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356784 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.356845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356807 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.356845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356830 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.357040 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356904 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.357040 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356947 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8wh9\" (UniqueName: \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-kube-api-access-l8wh9\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.357040 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.356987 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.357226 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.357087 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.357226 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.357137 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.357226 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.357171 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.357226 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.357209 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.357441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.357224 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.357441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.357248 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc7ns\" (UniqueName: \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-kube-api-access-sc7ns\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.357441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.357307 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.357441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.357408 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.359051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.359024 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.359511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.359495 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.369592 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.369575 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.369683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.369663 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc7ns\" (UniqueName: \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-kube-api-access-sc7ns\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.458954 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.458878 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.458954 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.458916 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.458954 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.458938 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459232 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.458987 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459232 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.459011 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459232 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.459053 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8wh9\" (UniqueName: \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-kube-api-access-l8wh9\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459232 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.459086 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.459251 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.459282 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.459394 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.459448 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459732 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.459689 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459732 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.459719 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.459854 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.459739 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.461946 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.461686 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.463294 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.462519 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.466250 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.466226 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht"] Oct 31 13:13:52.467263 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:52.467242 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d057adb_f72c_4327_b979_aa61fdf98068.slice/crio-6037b0b2c959e5218a3d88241997d4869e4bb93df8b68c62d98ee755d8d2476e WatchSource:0}: Error finding container 6037b0b2c959e5218a3d88241997d4869e4bb93df8b68c62d98ee755d8d2476e: Status 404 returned error can't find the container with id 6037b0b2c959e5218a3d88241997d4869e4bb93df8b68c62d98ee755d8d2476e Oct 31 13:13:52.469770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.469733 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:52.469889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.469796 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:52.469889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.469824 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:52.470406 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.470336 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:52.472800 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.472778 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8wh9\" (UniqueName: \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-kube-api-access-l8wh9\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.473855 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.473832 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.584576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.584543 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:52.610773 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.606628 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" event={"ID":"2d057adb-f72c-4327-b979-aa61fdf98068","Type":"ContainerStarted","Data":"b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645"} Oct 31 13:13:52.610773 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.606705 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" event={"ID":"2d057adb-f72c-4327-b979-aa61fdf98068","Type":"ContainerStarted","Data":"6037b0b2c959e5218a3d88241997d4869e4bb93df8b68c62d98ee755d8d2476e"} Oct 31 13:13:52.636208 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.636127 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" podStartSLOduration=1.6361053559999998 podStartE2EDuration="1.636105356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:52.630840619 +0000 UTC m=+849.476007106" watchObservedRunningTime="2026-10-31 13:13:52.636105356 +0000 UTC m=+849.481271845" Oct 31 13:13:52.763223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.763188 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb"] Oct 31 13:13:52.765588 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:52.765559 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d85d750_b72f_4664_b3b2_1a0f802a6d81.slice/crio-b6906c5c53c4f60876595610667f275091c3cd1af67ef33acaf2e59d02a89878 WatchSource:0}: Error finding container b6906c5c53c4f60876595610667f275091c3cd1af67ef33acaf2e59d02a89878: Status 404 returned error can't find the container with id b6906c5c53c4f60876595610667f275091c3cd1af67ef33acaf2e59d02a89878 Oct 31 13:13:52.768302 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.768265 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:52.768391 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.768345 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:52.768391 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.768384 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:52.889955 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.889336 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv"] Oct 31 13:13:52.890412 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:13:52.890383 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb098adeb_3a2f_479b_a72c_93bb60f4bba9.slice/crio-17fe5fcd3b1ff01eb074cee65466b22249ca0dc5aa601b01d020d684a8764b43 WatchSource:0}: Error finding container 17fe5fcd3b1ff01eb074cee65466b22249ca0dc5aa601b01d020d684a8764b43: Status 404 returned error can't find the container with id 17fe5fcd3b1ff01eb074cee65466b22249ca0dc5aa601b01d020d684a8764b43 Oct 31 13:13:52.893823 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.893781 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:52.893959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.893869 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:52.893959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:52.893917 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:13:53.176870 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:53.176837 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:13:53.178612 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:53.178575 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 13:13:53.178751 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:53.178625 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 13:13:53.623525 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:53.622689 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" event={"ID":"3d85d750-b72f-4664-b3b2-1a0f802a6d81","Type":"ContainerStarted","Data":"b1127fccc9168a3056055ebb0fee7b34e48ff64e03e23eba1cea601d572fdab6"} Oct 31 13:13:53.623525 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:53.622770 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" event={"ID":"3d85d750-b72f-4664-b3b2-1a0f802a6d81","Type":"ContainerStarted","Data":"b6906c5c53c4f60876595610667f275091c3cd1af67ef33acaf2e59d02a89878"} Oct 31 13:13:53.625255 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:53.625214 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" event={"ID":"b098adeb-3a2f-479b-a72c-93bb60f4bba9","Type":"ContainerStarted","Data":"24c4b6923d4d42555c1aeeff1e525e48666458f4a9c45a6b9d8844c967554634"} Oct 31 13:13:53.625391 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:53.625271 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" event={"ID":"b098adeb-3a2f-479b-a72c-93bb60f4bba9","Type":"ContainerStarted","Data":"17fe5fcd3b1ff01eb074cee65466b22249ca0dc5aa601b01d020d684a8764b43"} Oct 31 13:13:53.716533 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:53.706421 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" podStartSLOduration=1.706400694 podStartE2EDuration="1.706400694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:53.672199482 +0000 UTC m=+850.517365900" watchObservedRunningTime="2026-10-31 13:13:53.706400694 +0000 UTC m=+850.551567114" Oct 31 13:13:53.726798 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:53.726737 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" podStartSLOduration=1.726703224 podStartE2EDuration="1.726703224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:53.714944412 +0000 UTC m=+850.560110837" watchObservedRunningTime="2026-10-31 13:13:53.726703224 +0000 UTC m=+850.571869643" Oct 31 13:13:53.928895 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:13:53.928173 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:13:54.181558 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:54.181153 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 13:13:54.181558 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:54.181218 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 13:13:54.490556 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:54.489917 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:13:54.521916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:54.521613 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 13:13:54.521916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:54.521683 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 13:13:54.601183 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:54.600233 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:13:54.612913 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:54.612624 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 13:13:54.612913 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:54.612707 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 13:13:54.910505 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:54.907850 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:54.932292 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:54.932264 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:55.178556 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:55.177794 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 13:13:55.178556 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:55.177871 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 13:13:55.482559 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:55.471815 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 13:13:55.482559 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:55.471878 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 13:13:55.598404 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:55.596284 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 13:13:55.622650 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:55.622601 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 13:13:55.729163 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:55.729138 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:13:56.202500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:56.199364 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 13:13:56.202500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:56.199433 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 13:13:56.478153 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:56.477493 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 13:13:56.478153 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:56.477556 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 13:13:56.621321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:56.620868 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 13:13:56.621321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:56.620929 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 13:13:57.222752 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:57.222719 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 13:13:57.223355 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:57.223323 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 13:13:57.475918 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:57.475819 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 13:13:57.476161 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:57.476131 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 13:13:57.614681 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:57.614634 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 13:13:57.615000 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:57.614950 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 13:13:58.188049 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:58.188012 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 13:13:58.191583 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:58.191541 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 13:13:58.508562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:58.503146 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 13:13:58.508562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:58.503234 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 13:13:58.585865 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:58.585824 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 13:13:58.586164 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:58.586135 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 13:13:59.180724 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.177119 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 13:13:59.180724 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.177198 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 13:13:59.471713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.471618 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 13:13:59.471889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.471683 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 13:13:59.493253 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.493207 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6"] Oct 31 13:13:59.501431 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.501406 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.511312 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.511281 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6"] Oct 31 13:13:59.520653 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.520401 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-79c6ff\"/\"distinct-2-w6-79c6ff-kserve-self-signed-certs\"" Oct 31 13:13:59.550754 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.550724 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-model-cache\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.550978 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.550958 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-home\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.551121 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.551094 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-dshm\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.551209 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.551158 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-tmp-dir\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.551280 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.551264 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03559efd-42ab-4720-9111-a230f6c30c41-tls-certs\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.551327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.551312 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmj2b\" (UniqueName: \"kubernetes.io/projected/03559efd-42ab-4720-9111-a230f6c30c41-kube-api-access-zmj2b\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.585233 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.585196 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 13:13:59.585407 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.585317 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 13:13:59.653397 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.653358 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03559efd-42ab-4720-9111-a230f6c30c41-tls-certs\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.654167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.654141 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmj2b\" (UniqueName: \"kubernetes.io/projected/03559efd-42ab-4720-9111-a230f6c30c41-kube-api-access-zmj2b\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.654790 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.654767 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-model-cache\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.655122 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.655102 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-home\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.655935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.655917 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-dshm\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.656398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.656380 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-tmp-dir\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.656679 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.655626 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-model-cache\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.656807 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.655868 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-home\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.657296 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.657274 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-tmp-dir\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.660738 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.660646 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-dshm\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.663912 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.663888 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03559efd-42ab-4720-9111-a230f6c30c41-tls-certs\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.668375 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.668347 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmj2b\" (UniqueName: \"kubernetes.io/projected/03559efd-42ab-4720-9111-a230f6c30c41-kube-api-access-zmj2b\") pod \"distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:13:59.826326 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:13:59.826300 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:14:00.156168 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.156143 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6"] Oct 31 13:14:00.461158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.458450 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:14:00.461158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.460340 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:14:00.466107 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.464368 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:14:00.640985 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.640957 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:14:00.643105 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.642991 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:14:00.650689 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.650667 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:14:00.836140 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.836099 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" event={"ID":"03559efd-42ab-4720-9111-a230f6c30c41","Type":"ContainerStarted","Data":"0191a9bf51a7e7763c4841c141ea03c319fa07e05577e9d48ff8b9a4b9fb23e3"} Oct 31 13:14:00.836328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.836157 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" event={"ID":"03559efd-42ab-4720-9111-a230f6c30c41","Type":"ContainerStarted","Data":"a753e877b612535be5308c218a011ee9abb1476648f9226df6a1eb96c5b3d3f5"} Oct 31 13:14:00.856749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.856703 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podStartSLOduration=1.85668815 podStartE2EDuration="1.85668815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:00.85588871 +0000 UTC m=+857.701055126" watchObservedRunningTime="2026-10-31 13:14:00.85668815 +0000 UTC m=+857.701854568" Oct 31 13:14:00.911131 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.911101 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:14:00.911532 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.911509 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:14:00.913207 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:00.913174 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:14:01.826765 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:01.826738 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:14:07.080493 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.080445 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw"] Oct 31 13:14:07.084684 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.084661 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.105799 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.104750 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw"] Oct 31 13:14:07.105799 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.104819 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv"] Oct 31 13:14:07.109772 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.109593 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.136818 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.136778 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv"] Oct 31 13:14:07.178894 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.178858 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c"] Oct 31 13:14:07.182967 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.182948 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.199676 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c"] Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.199901 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.199943 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200009 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200038 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75s6v\" (UniqueName: \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-kube-api-access-75s6v\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200126 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200149 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200180 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f067868a-a184-4ef4-9f5c-7de384be46ae-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200232 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200260 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200443 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200543 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.200597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200569 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.201223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200648 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg67m\" (UniqueName: \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-kube-api-access-xg67m\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.201223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200702 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.201223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200765 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.201223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200791 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.201223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200828 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.201223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.200894 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.302132 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302092 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwlxt\" (UniqueName: \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-kube-api-access-vwlxt\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.302337 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302159 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.302337 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302191 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.302337 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302232 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02d8eed-d690-4d65-981a-03b5a4331852-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.302337 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302285 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.302597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302368 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75s6v\" (UniqueName: \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-kube-api-access-75s6v\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.302597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302578 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.302705 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302599 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.302705 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302643 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.302705 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302682 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.302862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302703 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.302862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302773 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.302862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302811 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f067868a-a184-4ef4-9f5c-7de384be46ae-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.302862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302854 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.303061 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302884 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.303061 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302915 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.303061 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302961 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.303061 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.302979 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.303061 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303034 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.303305 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303073 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.303305 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303115 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.303305 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303145 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.303305 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303179 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg67m\" (UniqueName: \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-kube-api-access-xg67m\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.303305 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303209 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02d8eed-d690-4d65-981a-03b5a4331852-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.303305 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303263 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.303638 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303345 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.303638 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303372 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.303638 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303423 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.304881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.303781 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.304881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.304008 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.304881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.304071 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f067868a-a184-4ef4-9f5c-7de384be46ae-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.304881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.304097 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.304881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.304321 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.304881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.304327 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.304881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.304419 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.304881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.304464 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.304881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.304806 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.305905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.305846 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.306151 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.306128 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.307314 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.307295 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.307710 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.307691 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.318555 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.318529 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75s6v\" (UniqueName: \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-kube-api-access-75s6v\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.319522 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.318879 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.319647 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.319627 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.319811 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.319795 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg67m\" (UniqueName: \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-kube-api-access-xg67m\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.406018 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.405922 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406018 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.405982 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406234 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406034 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406234 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406131 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406234 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406161 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406234 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406207 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02d8eed-d690-4d65-981a-03b5a4331852-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406434 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406335 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406434 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406399 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwlxt\" (UniqueName: \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-kube-api-access-vwlxt\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406577 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406460 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02d8eed-d690-4d65-981a-03b5a4331852-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406577 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406507 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406577 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406505 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406769 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406745 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.406905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.406885 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.407140 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.407119 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02d8eed-d690-4d65-981a-03b5a4331852-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.408145 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.408128 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:07.408383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.408366 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.408861 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.408843 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02d8eed-d690-4d65-981a-03b5a4331852-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.428567 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.428539 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwlxt\" (UniqueName: \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-kube-api-access-vwlxt\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.428741 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.428723 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.452550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.452528 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:07.502046 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.502001 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:07.704788 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.704758 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw"] Oct 31 13:14:07.705585 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:14:07.705559 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabf6be0_70ac_4a23_bce4_23a0047c0cfc.slice/crio-25cb2e4d0248f06bd82effb470cc81f252c675bcf692a1fa713a124e0f139d37 WatchSource:0}: Error finding container 25cb2e4d0248f06bd82effb470cc81f252c675bcf692a1fa713a124e0f139d37: Status 404 returned error can't find the container with id 25cb2e4d0248f06bd82effb470cc81f252c675bcf692a1fa713a124e0f139d37 Oct 31 13:14:07.708245 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.708209 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:07.708338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.708271 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:07.708338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.708302 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:07.761290 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.760984 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv"] Oct 31 13:14:07.761400 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:14:07.761338 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf067868a_a184_4ef4_9f5c_7de384be46ae.slice/crio-f339e3ea00dbac319a0a912ad092b8bec274c0abc98a490bba15d62a53ca571a WatchSource:0}: Error finding container f339e3ea00dbac319a0a912ad092b8bec274c0abc98a490bba15d62a53ca571a: Status 404 returned error can't find the container with id f339e3ea00dbac319a0a912ad092b8bec274c0abc98a490bba15d62a53ca571a Oct 31 13:14:07.763996 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.763965 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:07.764076 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.764028 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:07.764076 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.764057 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:07.804389 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.804357 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c"] Oct 31 13:14:07.817337 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.817127 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:07.818437 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.818394 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:07.818744 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.818683 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:07.898750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.890865 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" event={"ID":"e02d8eed-d690-4d65-981a-03b5a4331852","Type":"ContainerStarted","Data":"c878a82638ce23653ed7f997cd46b50a04e781717ea53ac82b2238b487ddbddc"} Oct 31 13:14:07.898750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.893026 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" event={"ID":"f067868a-a184-4ef4-9f5c-7de384be46ae","Type":"ContainerStarted","Data":"052a9350b364d89b9bd5b10c3f565122db1d1d0f0239d00d69c450b7a8689f3f"} Oct 31 13:14:07.898750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.893081 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" event={"ID":"f067868a-a184-4ef4-9f5c-7de384be46ae","Type":"ContainerStarted","Data":"f339e3ea00dbac319a0a912ad092b8bec274c0abc98a490bba15d62a53ca571a"} Oct 31 13:14:07.898750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.895965 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" event={"ID":"eabf6be0-70ac-4a23-bce4-23a0047c0cfc","Type":"ContainerStarted","Data":"59e673e9746826f22d78396ca06993de0da31eaa0ce54c897e25aaa03783246d"} Oct 31 13:14:07.898750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.896002 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" event={"ID":"eabf6be0-70ac-4a23-bce4-23a0047c0cfc","Type":"ContainerStarted","Data":"25cb2e4d0248f06bd82effb470cc81f252c675bcf692a1fa713a124e0f139d37"} Oct 31 13:14:07.922563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.922470 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podStartSLOduration=0.922452211 podStartE2EDuration="922.452211ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:07.921333361 +0000 UTC m=+864.766499770" watchObservedRunningTime="2026-10-31 13:14:07.922452211 +0000 UTC m=+864.767618630" Oct 31 13:14:07.949214 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:07.949167 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podStartSLOduration=0.949152101 podStartE2EDuration="949.152101ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:07.945331699 +0000 UTC m=+864.790498106" watchObservedRunningTime="2026-10-31 13:14:07.949152101 +0000 UTC m=+864.794318519" Oct 31 13:14:08.408711 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:08.408674 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:08.410417 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:08.410389 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:08.410567 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:08.410446 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:08.453407 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:08.453377 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:08.455050 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:08.455020 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:08.455206 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:08.455079 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:08.954501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:08.939273 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" event={"ID":"e02d8eed-d690-4d65-981a-03b5a4331852","Type":"ContainerStarted","Data":"0af2b04ccc271ab6f25bf533782c5535b14cb83381f42aa675437ce156786c87"} Oct 31 13:14:09.023590 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:09.011105 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podStartSLOduration=2.011085392 podStartE2EDuration="2.011085392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:09.008234235 +0000 UTC m=+865.853400654" watchObservedRunningTime="2026-10-31 13:14:09.011085392 +0000 UTC m=+865.856251813" Oct 31 13:14:09.415051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:09.413954 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:09.415051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:09.414060 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:09.462502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:09.462164 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:09.462502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:09.462224 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:09.517498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:09.512753 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:09.517498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:09.516894 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:09.517498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:09.516971 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:09.829717 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:09.829672 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:14:10.027354 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:10.027326 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:14:10.063501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:10.062849 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:14:10.417506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:10.415589 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:10.417506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:10.415651 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:10.465503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:10.463756 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:10.465503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:10.463812 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:10.521575 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:10.518869 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:10.521575 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:10.518929 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:11.416520 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:11.414742 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:11.416520 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:11.414809 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:11.461500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:11.461216 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:11.461500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:11.461278 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:11.513501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:11.509108 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:11.513501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:11.509173 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:12.179841 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.179788 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:12.184572 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.184531 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:12.412017 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.411817 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:12.412494 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.412285 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:12.423627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.423589 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:12.496412 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.460830 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:12.497043 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.496993 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:12.549685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.549649 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:12.574630 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.574579 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:12.598497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.586893 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 13:14:12.633849 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.619600 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:14:12.718501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.717721 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:12.774218 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.774047 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:14:12.810634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.810581 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:12.892859 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.892811 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:12.915190 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:12.915134 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:13.144707 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.144645 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:13.191511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.187565 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:13.191511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.189392 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:14:13.209617 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.209570 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:13.211709 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.211684 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:14:13.435557 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.422362 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:13.435557 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.422420 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:13.526518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.525571 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:14:13.584506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.528292 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:13.584506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.528345 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:13.676503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.652577 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 13:14:13.677346 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.653109 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:14:13.678322 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.653427 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:13.678918 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.678880 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:13.719765 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.719509 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:14:13.778498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.775656 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:13.823055 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.823005 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 13:14:13.899834 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.899787 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:14:13.935740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.929645 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:14:13.935740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.929710 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:14:13.935740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.929752 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:14:13.935740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.929787 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:14:13.935740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.929819 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:14:13.935740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.929852 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:14:13.935740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.929884 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:14:13.964896 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.964850 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 13:14:13.965608 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:13.965585 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:14:14.169543 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.163608 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:14:14.169543 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.164234 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:14:14.189611 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.189566 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:14.216622 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.216575 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:14.489713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.489621 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:14.489973 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.489937 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:14.491353 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.491326 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:14.863510 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.861278 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:14.909161 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.854544 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:14:14.910761 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.862828 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 13:14:14.918600 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.875668 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" podUID="65e8dcbf-3370-4090-a923-cd11a92a5749" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.19:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:14.918792 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.900285 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:14.918952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.918923 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:14.919068 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.907842 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 13:14:14.931500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.923943 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:14:14.931500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.923566 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:14.931734 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.854063 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:14.946738 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.946702 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:14.947038 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:14.946993 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:15.027827 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.027694 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:15.091533 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.091456 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:14:15.165697 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.164646 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:15.227203 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.227146 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:15.254995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.254941 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 31 13:14:15.409536 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.277580 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:15.409834 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.409802 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:15.433425 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.433303 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:15.433683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.433644 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:15.462791 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.462744 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:15.485504 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.463801 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:15.570835 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.560987 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:15.570835 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.561044 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:15.929501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.929393 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:14:15.930062 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.929801 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 31 13:14:15.930062 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.929915 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:15.938513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.931357 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:15.938513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.931433 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 13:14:15.938513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.931985 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:14:15.963723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.953872 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:15.963723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.953938 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.035907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.975716 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:16.035907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.979595 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.035907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:15.979689 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.036713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.036302 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.043547 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.040060 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:16.043547 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.040114 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.043547 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.040202 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.070502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.066555 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.194714 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.193658 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:14:16.210535 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.198108 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:14:16.295531 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.242688 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 13:14:16.390524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.373697 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:16.390524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.373764 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.420302 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.418648 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 31 13:14:16.457659 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.457565 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:16.485499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.461553 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:16.485499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.457762 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:16.485499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.484624 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:16.536502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.528396 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:16.536502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.528456 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:16.621670 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.621620 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:16.628515 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.621969 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.651596 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.634565 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" podUID="65e8dcbf-3370-4090-a923-cd11a92a5749" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.19:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.761432 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.761338 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:16.761731 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.761698 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.932278 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.932234 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.952703 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.932839 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.953975 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.932873 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.954816 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.952003 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.955422 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.952046 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.974134 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.974099 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:16.981591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:16.981558 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:14:17.072800 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.072758 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:17.206686 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.202615 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:14:17.206686 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.202987 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:14:17.251852 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.251807 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:17.333185 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.333073 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:17.333510 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.333448 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:17.415000 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.414965 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:17.420526 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.415260 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:17.420818 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.420794 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 31 13:14:17.491185 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.491148 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:17.491546 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.491425 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:17.594558 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.550602 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:17.594558 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.567605 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:17.594558 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.567666 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:17.853387 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.853272 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 31 13:14:17.959626 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.959576 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:17.962448 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.962412 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:17.968664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.967691 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:17.969354 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.968853 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:14:17.995500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.971077 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:18.019100 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:17.999651 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:18.086502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.075563 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:18.090938 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.090909 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:14:18.111279 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.111182 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:18.209638 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.209594 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:14:18.210300 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.210266 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:14:18.317581 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.317540 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 13:14:18.444562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.431243 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:18.444562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.431317 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:18.486946 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.486912 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:18.487188 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.487157 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:18.510330 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.510293 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:18.510707 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.510670 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:18.963788 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.963747 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:14:18.985917 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.970635 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:18.987452 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.970671 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 13:14:19.019501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:18.970554 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 13:14:19.020151 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.019746 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:14:19.020574 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.019787 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:14:19.081623 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.081564 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:19.214393 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.214216 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:19.214639 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.214596 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:14:19.336493 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.336320 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:19.421509 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.421441 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:19.421694 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.421529 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:19.513551 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.491963 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 31 13:14:19.522930 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.522679 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:19.523191 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.523162 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:19.531263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.531236 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:19.531442 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.531419 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:19.618170 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.618134 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:14:19.809566 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.807929 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded" Oct 31 13:14:19.910595 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.909631 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:14:19.911381 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.911361 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:14:19.993806 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.993177 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:19.994320 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:19.993828 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:14:20.019808 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.018810 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:14:20.026063 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.026001 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 13:14:20.068166 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.068139 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:14:20.076734 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.076193 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:20.090713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.090657 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:20.095128 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.095103 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:14:20.113514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.112678 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:14:20.126894 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.120736 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:14:20.187101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.187076 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:14:20.410131 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.410043 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 13:14:20.410131 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.410112 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 13:14:20.453657 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.453628 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:14:20.453804 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.453682 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 13:14:20.503639 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.503602 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 13:14:20.503845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:20.503668 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 13:14:21.552075 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:21.552045 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:21.552677 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:21.552655 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:21.553979 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:21.553958 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:14:22.409285 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:22.409257 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:22.450637 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:22.450588 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:22.450920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:22.450877 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:22.458764 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:22.458732 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:22.472365 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:22.472285 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:22.500659 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:22.500608 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:22.500808 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:22.500693 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:22.717046 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:22.716954 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:23.210421 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:23.210391 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:14:23.210667 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:23.210653 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:14:23.832912 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:14:23.832877 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:14:36.678193 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.678162 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t"] Oct 31 13:14:36.682353 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.682336 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.698896 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.698834 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t"] Oct 31 13:14:36.792096 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.792069 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.792096 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.792100 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.792293 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.792229 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.792293 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.792260 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.792370 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.792357 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.792408 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.792384 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.792450 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.792439 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.792546 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.792529 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9z9\" (UniqueName: \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-kube-api-access-7q9z9\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.792604 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.792594 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.842108 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.842077 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq"] Oct 31 13:14:36.847399 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.847376 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:36.863579 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.863552 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq"] Oct 31 13:14:36.894196 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.894166 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.894423 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.894394 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.894565 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.894531 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.894615 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.894580 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.894662 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.894644 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.894719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.894700 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9z9\" (UniqueName: \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-kube-api-access-7q9z9\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.894799 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.894783 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.894854 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.894834 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.895181 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.895160 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.895412 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.895393 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.895553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.895433 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.895553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.895514 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.895676 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.895583 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.895831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.895805 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.897837 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.897811 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.898237 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.898217 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.915659 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.915637 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9z9\" (UniqueName: \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-kube-api-access-7q9z9\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.915742 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.915686 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.997357 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.997282 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:36.997513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.997367 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:36.997569 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.997522 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:36.997610 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.997567 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:36.997670 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.997651 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:36.997721 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.997699 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:36.997833 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.997813 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrz94\" (UniqueName: \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-kube-api-access-rrz94\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:36.997880 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.997852 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:36.997986 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.997971 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:36.998030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:36.998023 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6150f29-b21c-4142-8b99-7fb0af958d0d-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.006394 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.006364 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq"] Oct 31 13:14:37.010357 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.010342 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.030916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.030888 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq"] Oct 31 13:14:37.100052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.099720 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.100052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.099787 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.100052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.099813 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.100052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.099845 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.100052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.099898 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrz94\" (UniqueName: \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-kube-api-access-rrz94\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.100052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.099932 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.100052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.099962 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.100052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.099995 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.100052 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.100036 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.100635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.100071 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.100635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.100119 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkxc\" (UniqueName: \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-kube-api-access-fvkxc\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.100635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.100164 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.100635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.100216 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.100635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.100257 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6150f29-b21c-4142-8b99-7fb0af958d0d-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.100635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.100301 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.100635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.100401 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.100635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.100461 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.100635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.100556 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.102345 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.101062 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.102345 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.101953 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.102345 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.102017 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6150f29-b21c-4142-8b99-7fb0af958d0d-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.102345 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.102119 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.102603 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.102511 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.105116 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.103158 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.105116 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.104045 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.116156 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.116128 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.116778 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.116755 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrz94\" (UniqueName: \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-kube-api-access-rrz94\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.168577 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.168552 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:37.202077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202046 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.202212 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202087 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.202212 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202159 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.202344 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202216 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.202344 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202241 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.202448 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202335 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.202448 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202388 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.202448 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202439 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkxc\" (UniqueName: \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-kube-api-access-fvkxc\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.202649 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202531 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.202649 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202568 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.202943 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202923 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.203024 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.202989 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.203082 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.203039 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.203139 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.203103 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.204957 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.204937 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.205588 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.205560 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.215638 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.215605 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.216611 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.216590 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkxc\" (UniqueName: \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-kube-api-access-fvkxc\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.308319 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.308246 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t"] Oct 31 13:14:37.312610 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:14:37.312583 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de77bc5_25f8_4e11_ba04_f3f254a594fc.slice/crio-4595479e37229430335b6cb70e85411322a748c99ecfbc450e4a41cf4796ec23 WatchSource:0}: Error finding container 4595479e37229430335b6cb70e85411322a748c99ecfbc450e4a41cf4796ec23: Status 404 returned error can't find the container with id 4595479e37229430335b6cb70e85411322a748c99ecfbc450e4a41cf4796ec23 Oct 31 13:14:37.315128 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.315087 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:37.315232 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.315186 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:37.315300 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.315257 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:37.365259 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.364997 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:37.468649 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.467993 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq"] Oct 31 13:14:37.469900 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:14:37.469870 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6150f29_b21c_4142_8b99_7fb0af958d0d.slice/crio-ee72f05c9cae828b29886e57028aaf68e55a9272027f477122f9ff721466a34d WatchSource:0}: Error finding container ee72f05c9cae828b29886e57028aaf68e55a9272027f477122f9ff721466a34d: Status 404 returned error can't find the container with id ee72f05c9cae828b29886e57028aaf68e55a9272027f477122f9ff721466a34d Oct 31 13:14:37.477630 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.475406 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:37.477630 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.475525 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:37.477630 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.475610 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:37.678279 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.678241 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq"] Oct 31 13:14:37.679416 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:14:37.679385 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68e7546_4bb1_4dd6_8f71_2d3c82c9e9c0.slice/crio-5f583486424f88365bd71b4861246a677d9460e3be51d270ca3f0570677ea215 WatchSource:0}: Error finding container 5f583486424f88365bd71b4861246a677d9460e3be51d270ca3f0570677ea215: Status 404 returned error can't find the container with id 5f583486424f88365bd71b4861246a677d9460e3be51d270ca3f0570677ea215 Oct 31 13:14:37.682986 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.682946 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:37.683067 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.683055 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:37.683168 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:37.683148 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:14:38.321828 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.321791 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" event={"ID":"2de77bc5-25f8-4e11-ba04-f3f254a594fc","Type":"ContainerStarted","Data":"e2880caaec7243ba4dc57ce5d35c28baf70bf20b1d5573aa8cc2f26eb9d1ddd3"} Oct 31 13:14:38.322015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.321853 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" event={"ID":"2de77bc5-25f8-4e11-ba04-f3f254a594fc","Type":"ContainerStarted","Data":"4595479e37229430335b6cb70e85411322a748c99ecfbc450e4a41cf4796ec23"} Oct 31 13:14:38.323545 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.323512 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" event={"ID":"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0","Type":"ContainerStarted","Data":"299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3"} Oct 31 13:14:38.323689 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.323565 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" event={"ID":"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0","Type":"ContainerStarted","Data":"5f583486424f88365bd71b4861246a677d9460e3be51d270ca3f0570677ea215"} Oct 31 13:14:38.325107 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.325063 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" event={"ID":"d6150f29-b21c-4142-8b99-7fb0af958d0d","Type":"ContainerStarted","Data":"b48ebd6102c9beb09791a08c40b5fd062244248748fcf5d514de4ea0671637ed"} Oct 31 13:14:38.325210 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.325125 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" event={"ID":"d6150f29-b21c-4142-8b99-7fb0af958d0d","Type":"ContainerStarted","Data":"ee72f05c9cae828b29886e57028aaf68e55a9272027f477122f9ff721466a34d"} Oct 31 13:14:38.385613 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.380883 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:38.397404 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.391630 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:14:38.397404 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.391694 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 13:14:38.402362 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.402304 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" podStartSLOduration=2.402287546 podStartE2EDuration="2.402287546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:38.365904575 +0000 UTC m=+895.211070994" watchObservedRunningTime="2026-10-31 13:14:38.402287546 +0000 UTC m=+895.247453963" Oct 31 13:14:38.402801 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.402777 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" podStartSLOduration=2.402768548 podStartE2EDuration="2.402768548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:38.399853075 +0000 UTC m=+895.245019494" watchObservedRunningTime="2026-10-31 13:14:38.402768548 +0000 UTC m=+895.247934963" Oct 31 13:14:38.429484 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:38.429386 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" podStartSLOduration=2.429371606 podStartE2EDuration="2.429371606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:38.426577516 +0000 UTC m=+895.271743934" watchObservedRunningTime="2026-10-31 13:14:38.429371606 +0000 UTC m=+895.274538024" Oct 31 13:14:39.006921 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:39.006881 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:39.011630 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:39.011596 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 13:14:39.011881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:39.011855 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 13:14:39.186501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:39.185659 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:39.186501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:39.185804 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 13:14:39.186501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:39.185852 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 13:14:39.369322 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:39.369279 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:14:39.369657 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:39.369624 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 13:14:40.021425 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:40.011859 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 13:14:40.021425 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:40.011925 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 13:14:40.173909 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:40.173869 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 13:14:40.174193 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:40.174163 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 13:14:40.387271 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:40.387226 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:14:40.387507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:40.387292 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 13:14:40.999544 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:40.998869 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 13:14:40.999544 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:40.998934 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 13:14:41.211166 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:41.211123 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 13:14:41.219537 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:41.215601 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 13:14:41.380496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:41.379907 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:14:41.380496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:41.379971 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 13:14:42.031652 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:42.028430 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 13:14:42.031652 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:42.028507 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 13:14:42.187501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:42.183566 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 13:14:42.187501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:42.183635 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 13:14:42.378498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:42.377558 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:14:42.378498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:42.377641 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 13:14:43.011501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:43.010831 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 13:14:43.011501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:43.010911 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 13:14:43.187501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:43.184096 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 13:14:43.187501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:43.184156 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 13:14:43.389501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:43.388643 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:14:43.389501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:43.388742 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 13:14:43.625813 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:43.625781 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:14:43.628143 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:43.628121 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:14:43.999091 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:43.999057 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 13:14:43.999288 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:43.999113 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 13:14:44.173495 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:44.169804 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 13:14:44.173495 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:44.169863 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 13:14:44.369873 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:44.369822 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:14:44.370084 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:44.369893 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 13:14:44.681809 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:44.681717 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:14:45.174370 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:45.174339 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:45.175685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:45.175662 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:45.179207 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:45.179188 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:14:45.468796 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:45.468700 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:45.469772 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:45.469703 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:45.470902 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:45.470875 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:14:45.650341 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:45.650305 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:45.651926 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:45.651899 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:45.652778 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:45.652755 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:14:47.878620 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:47.878568 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:47.956503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:47.945646 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:47.980009 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:47.979961 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:48.008243 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:47.980260 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:48.031759 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.031708 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:14:48.033430 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.033396 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:14:48.047536 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.034265 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:48.061502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.050331 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:48.135559 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.129674 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 31 13:14:48.161179 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.161127 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:48.188207 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.188088 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:48.246169 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.246126 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:48.446356 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.446236 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 31 13:14:48.960150 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.957877 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:48.961596 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:48.961574 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:14:49.019586 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.015608 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:14:49.019586 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.016389 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:14:49.057221 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.057169 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:14:49.258562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.154223 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:49.258562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.202665 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:14:49.258562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.202727 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:14:49.258562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.204423 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:49.258562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.204769 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:14:49.258562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.206403 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:49.258562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.234932 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:14:49.258562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.235266 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:14:49.258562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.236522 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 13:14:49.270627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.270582 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:49.321899 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.279783 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:14:49.329500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.325598 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:14:49.356624 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.356591 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:14:49.356959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.356938 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:14:49.621504 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.613939 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 31 13:14:49.868503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.820413 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded" Oct 31 13:14:49.905955 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.905858 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:14:49.906307 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.906280 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:14:49.981500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:49.968560 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:50.125546 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.125510 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:14:50.218676 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.218576 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:50.232507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.218936 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:50.233163 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.218990 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:14:50.236240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.219017 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 13:14:50.261496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.236583 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:14:50.264801 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.264760 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:14:50.295009 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.294969 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 13:14:50.479555 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.461105 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 13:14:50.648563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.622325 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:50.740091 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.739981 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:50.846640 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.846590 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:50.847229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.846888 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Oct 31 13:14:50.993561 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:50.992782 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:51.129502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.128580 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:14:51.236503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.235584 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:14:51.236703 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.236558 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:51.236785 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.236732 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:51.300995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.300880 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:14:51.345501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.342253 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:14:51.369660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.349852 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:51.370734 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.349891 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:51.470505 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.461938 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:51.633506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.631899 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:51.633720 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.633590 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qqtbc" podUID="65e8dcbf-3370-4090-a923-cd11a92a5749" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.19:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:51.994564 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:51.993621 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:52.138503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.131645 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:14:52.238513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.237586 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:52.242526 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.238853 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:14:52.242526 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.239136 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:14:52.341501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.321779 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:52.377412 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.377369 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:52.390840 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.390799 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:14:52.410677 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.398353 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:14:52.474603 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.474546 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:52.621889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.595577 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:14:52.621889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.595647 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:52.632691 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.632643 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:52.997754 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:52.997618 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:53.134988 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:53.134935 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:53.249553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:53.248611 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:53.249553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:53.249076 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 13:14:53.249553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:53.249423 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:14:53.329614 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:53.329565 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:53.385502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:53.380560 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:53.408733 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:53.406734 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:53.408733 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:53.407087 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:14:53.483987 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:53.483861 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:53.635612 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:53.635567 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:54.025581 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.015399 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:54.185954 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.168056 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:54.280646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.254239 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:14:54.280646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.254595 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:14:54.280646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.254899 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 13:14:54.344684 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.344641 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:54.385506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.379186 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:14:54.440494 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.423930 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:14:54.440494 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.424288 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:54.440494 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.431155 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:14:54.452492 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.441040 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:14:54.466502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.460543 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:14:54.497862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.494169 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:14:54.497862 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:14:54.495986 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:14:54.497862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.496890 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:14:54.554892 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.548191 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:14:54.570899 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.570845 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:14:54.574390 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.574362 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:14:54.577977 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.577956 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:14:54.578818 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.578801 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:14:54.579015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:14:54.579002 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:15:00.158586 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.158548 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr"] Oct 31 13:15:00.164536 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.164514 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" Oct 31 13:15:00.173999 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.173970 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr"] Oct 31 13:15:00.175666 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.175638 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\"" Oct 31 13:15:00.185570 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.185540 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng"] Oct 31 13:15:00.190419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.190397 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" Oct 31 13:15:00.205408 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.205378 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng"] Oct 31 13:15:00.268714 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.268683 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecdecd31-e480-4f1d-86c4-a023cc2e418e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr\" (UID: \"ecdecd31-e480-4f1d-86c4-a023cc2e418e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" Oct 31 13:15:00.268868 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.268746 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng\" (UID: \"98a79ba0-5690-4754-8ad0-8689bcaa1ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" Oct 31 13:15:00.268929 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.268861 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw48\" (UniqueName: \"kubernetes.io/projected/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-kube-api-access-cfw48\") pod \"maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng\" (UID: \"98a79ba0-5690-4754-8ad0-8689bcaa1ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" Oct 31 13:15:00.269085 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.269057 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtknj\" (UniqueName: \"kubernetes.io/projected/ecdecd31-e480-4f1d-86c4-a023cc2e418e-kube-api-access-rtknj\") pod \"maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr\" (UID: \"ecdecd31-e480-4f1d-86c4-a023cc2e418e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" Oct 31 13:15:00.370970 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.370934 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecdecd31-e480-4f1d-86c4-a023cc2e418e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr\" (UID: \"ecdecd31-e480-4f1d-86c4-a023cc2e418e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" Oct 31 13:15:00.371145 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.371011 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng\" (UID: \"98a79ba0-5690-4754-8ad0-8689bcaa1ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" Oct 31 13:15:00.371145 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.371051 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw48\" (UniqueName: \"kubernetes.io/projected/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-kube-api-access-cfw48\") pod \"maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng\" (UID: \"98a79ba0-5690-4754-8ad0-8689bcaa1ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" Oct 31 13:15:00.371226 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.371149 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtknj\" (UniqueName: \"kubernetes.io/projected/ecdecd31-e480-4f1d-86c4-a023cc2e418e-kube-api-access-rtknj\") pod \"maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr\" (UID: \"ecdecd31-e480-4f1d-86c4-a023cc2e418e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" Oct 31 13:15:00.371752 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.371732 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecdecd31-e480-4f1d-86c4-a023cc2e418e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr\" (UID: \"ecdecd31-e480-4f1d-86c4-a023cc2e418e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" Oct 31 13:15:00.371799 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.371762 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng\" (UID: \"98a79ba0-5690-4754-8ad0-8689bcaa1ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" Oct 31 13:15:00.389509 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.389447 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtknj\" (UniqueName: \"kubernetes.io/projected/ecdecd31-e480-4f1d-86c4-a023cc2e418e-kube-api-access-rtknj\") pod \"maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr\" (UID: \"ecdecd31-e480-4f1d-86c4-a023cc2e418e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" Oct 31 13:15:00.389823 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.389805 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw48\" (UniqueName: \"kubernetes.io/projected/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-kube-api-access-cfw48\") pod \"maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng\" (UID: \"98a79ba0-5690-4754-8ad0-8689bcaa1ae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" Oct 31 13:15:00.486118 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.486036 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" Oct 31 13:15:00.514285 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.514254 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" Oct 31 13:15:00.777502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.777459 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr"] Oct 31 13:15:00.778890 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:15:00.778860 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdecd31_e480_4f1d_86c4_a023cc2e418e.slice/crio-17a0e8563438ca03f56b28f7c6a92624cc2bcafc4052719fb803e3fae6aef478 WatchSource:0}: Error finding container 17a0e8563438ca03f56b28f7c6a92624cc2bcafc4052719fb803e3fae6aef478: Status 404 returned error can't find the container with id 17a0e8563438ca03f56b28f7c6a92624cc2bcafc4052719fb803e3fae6aef478 Oct 31 13:15:00.855646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:00.855617 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng"] Oct 31 13:15:00.856635 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:15:00.856608 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a79ba0_5690_4754_8ad0_8689bcaa1ae8.slice/crio-4db5a8909751f68c6e09fa1f1c81c30988609a9705d331cd2b111f137b4c83d6 WatchSource:0}: Error finding container 4db5a8909751f68c6e09fa1f1c81c30988609a9705d331cd2b111f137b4c83d6: Status 404 returned error can't find the container with id 4db5a8909751f68c6e09fa1f1c81c30988609a9705d331cd2b111f137b4c83d6 Oct 31 13:15:01.122606 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:01.122571 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" event={"ID":"98a79ba0-5690-4754-8ad0-8689bcaa1ae8","Type":"ContainerStarted","Data":"4db5a8909751f68c6e09fa1f1c81c30988609a9705d331cd2b111f137b4c83d6"} Oct 31 13:15:01.123948 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:01.123918 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" event={"ID":"ecdecd31-e480-4f1d-86c4-a023cc2e418e","Type":"ContainerStarted","Data":"17a0e8563438ca03f56b28f7c6a92624cc2bcafc4052719fb803e3fae6aef478"} Oct 31 13:15:05.157069 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:05.157037 4542 generic.go:356] "Generic (PLEG): container finished" podID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerID="b9f1ce25f3c614d67e2a2697142f4169d5ff2364ed653f9ad961051004b10af4" exitCode=60 Oct 31 13:15:05.157551 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:05.157124 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" event={"ID":"ecdecd31-e480-4f1d-86c4-a023cc2e418e","Type":"ContainerDied","Data":"b9f1ce25f3c614d67e2a2697142f4169d5ff2364ed653f9ad961051004b10af4"} Oct 31 13:15:05.157551 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:05.157366 4542 scope.go:122] "RemoveContainer" containerID="b9f1ce25f3c614d67e2a2697142f4169d5ff2364ed653f9ad961051004b10af4" Oct 31 13:15:05.158860 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:05.158836 4542 generic.go:356] "Generic (PLEG): container finished" podID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerID="f61e9758a0f8f56110f9304ec4c46cd30e07b62da235a7f226befd0d35cd111e" exitCode=60 Oct 31 13:15:05.158985 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:05.158931 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" event={"ID":"98a79ba0-5690-4754-8ad0-8689bcaa1ae8","Type":"ContainerDied","Data":"f61e9758a0f8f56110f9304ec4c46cd30e07b62da235a7f226befd0d35cd111e"} Oct 31 13:15:05.159175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:05.159160 4542 scope.go:122] "RemoveContainer" containerID="f61e9758a0f8f56110f9304ec4c46cd30e07b62da235a7f226befd0d35cd111e" Oct 31 13:15:06.167522 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:06.167489 4542 generic.go:356] "Generic (PLEG): container finished" podID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerID="0dc79fb744a12255a5bd577ff041724fbb8f7803d020f7b6037da1a59fd91c81" exitCode=60 Oct 31 13:15:06.167920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:06.167569 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" event={"ID":"98a79ba0-5690-4754-8ad0-8689bcaa1ae8","Type":"ContainerDied","Data":"0dc79fb744a12255a5bd577ff041724fbb8f7803d020f7b6037da1a59fd91c81"} Oct 31 13:15:06.167920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:06.167630 4542 scope.go:122] "RemoveContainer" containerID="f61e9758a0f8f56110f9304ec4c46cd30e07b62da235a7f226befd0d35cd111e" Oct 31 13:15:06.167920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:06.167910 4542 scope.go:122] "RemoveContainer" containerID="0dc79fb744a12255a5bd577ff041724fbb8f7803d020f7b6037da1a59fd91c81" Oct 31 13:15:06.168263 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:06.168228 4542 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng_odh-ai-gateway-infra(98a79ba0-5690-4754-8ad0-8689bcaa1ae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" Oct 31 13:15:06.169826 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:06.169809 4542 generic.go:356] "Generic (PLEG): container finished" podID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerID="cf62772259eaf0887131141c27cc60ad3922afbb5f126216e8fe214ec5fd18c8" exitCode=60 Oct 31 13:15:06.169919 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:06.169895 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" event={"ID":"ecdecd31-e480-4f1d-86c4-a023cc2e418e","Type":"ContainerDied","Data":"cf62772259eaf0887131141c27cc60ad3922afbb5f126216e8fe214ec5fd18c8"} Oct 31 13:15:06.170129 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:06.170115 4542 scope.go:122] "RemoveContainer" containerID="cf62772259eaf0887131141c27cc60ad3922afbb5f126216e8fe214ec5fd18c8" Oct 31 13:15:06.170379 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:06.170362 4542 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr_odh-ai-gateway-infra(ecdecd31-e480-4f1d-86c4-a023cc2e418e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" Oct 31 13:15:06.180063 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:06.180047 4542 scope.go:122] "RemoveContainer" containerID="b9f1ce25f3c614d67e2a2697142f4169d5ff2364ed653f9ad961051004b10af4" Oct 31 13:15:07.178649 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:07.178619 4542 scope.go:122] "RemoveContainer" containerID="cf62772259eaf0887131141c27cc60ad3922afbb5f126216e8fe214ec5fd18c8" Oct 31 13:15:07.179128 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:07.179005 4542 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr_odh-ai-gateway-infra(ecdecd31-e480-4f1d-86c4-a023cc2e418e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" Oct 31 13:15:07.180243 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:07.180220 4542 scope.go:122] "RemoveContainer" containerID="0dc79fb744a12255a5bd577ff041724fbb8f7803d020f7b6037da1a59fd91c81" Oct 31 13:15:07.180601 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:07.180577 4542 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng_odh-ai-gateway-infra(98a79ba0-5690-4754-8ad0-8689bcaa1ae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" Oct 31 13:15:10.932518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:10.932464 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg"] Oct 31 13:15:19.285439 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:19.285405 4542 scope.go:122] "RemoveContainer" containerID="cf62772259eaf0887131141c27cc60ad3922afbb5f126216e8fe214ec5fd18c8" Oct 31 13:15:19.286002 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:19.285459 4542 scope.go:122] "RemoveContainer" containerID="0dc79fb744a12255a5bd577ff041724fbb8f7803d020f7b6037da1a59fd91c81" Oct 31 13:15:20.327095 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:20.327059 4542 generic.go:356] "Generic (PLEG): container finished" podID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerID="1e6cb818324e8a8600a83260cc9d730bd8bb7237f05d0337b810e178ca995fe4" exitCode=60 Oct 31 13:15:20.327634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:20.327128 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" event={"ID":"98a79ba0-5690-4754-8ad0-8689bcaa1ae8","Type":"ContainerDied","Data":"1e6cb818324e8a8600a83260cc9d730bd8bb7237f05d0337b810e178ca995fe4"} Oct 31 13:15:20.327634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:20.327188 4542 scope.go:122] "RemoveContainer" containerID="0dc79fb744a12255a5bd577ff041724fbb8f7803d020f7b6037da1a59fd91c81" Oct 31 13:15:20.327634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:20.327616 4542 scope.go:122] "RemoveContainer" containerID="1e6cb818324e8a8600a83260cc9d730bd8bb7237f05d0337b810e178ca995fe4" Oct 31 13:15:20.327992 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:20.327967 4542 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng_odh-ai-gateway-infra(98a79ba0-5690-4754-8ad0-8689bcaa1ae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" Oct 31 13:15:20.329359 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:20.329338 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" event={"ID":"ecdecd31-e480-4f1d-86c4-a023cc2e418e","Type":"ContainerStarted","Data":"a6dc479ffeb71d500fb549e60bc1cc0704d23583e929687ded417cacf9f73ad5"} Oct 31 13:15:20.368165 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:20.368119 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" podStartSLOduration=17.141585868 podStartE2EDuration="20.368105514s" totalImagesPullingTime="3.226519646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:20.365756192 +0000 UTC m=+937.210922612" watchObservedRunningTime="2026-10-31 13:15:20.368105514 +0000 UTC m=+937.213271932" Oct 31 13:15:21.661950 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:21.661909 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng"] Oct 31 13:15:21.677621 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:21.677587 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr"] Oct 31 13:15:21.801520 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:21.794802 4542 generic.go:356] "Generic (PLEG): container finished" podID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerID="a6dc479ffeb71d500fb549e60bc1cc0704d23583e929687ded417cacf9f73ad5" exitCode=60 Oct 31 13:15:21.801520 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:21.795015 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" event={"ID":"ecdecd31-e480-4f1d-86c4-a023cc2e418e","Type":"ContainerDied","Data":"a6dc479ffeb71d500fb549e60bc1cc0704d23583e929687ded417cacf9f73ad5"} Oct 31 13:15:21.801520 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:21.795081 4542 scope.go:122] "RemoveContainer" containerID="cf62772259eaf0887131141c27cc60ad3922afbb5f126216e8fe214ec5fd18c8" Oct 31 13:15:22.132107 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.128625 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs"] Oct 31 13:15:22.182502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.170084 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs"] Oct 31 13:15:22.182502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.170209 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.260639 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.260598 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:22.361498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.351512 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.361498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.357090 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.361498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.357123 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7c8d\" (UniqueName: \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-kube-api-access-s7c8d\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.361498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.357529 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.361498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.357569 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.361941 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.361838 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.414511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.413589 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2604e553-ae64-4fdf-ba10-1a639e5157f0-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.414511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.413853 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.414511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.413901 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.528526 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.507587 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 13:15:22.600501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.569018 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.600501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.569078 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.600501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.569265 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.600501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.569304 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2604e553-ae64-4fdf-ba10-1a639e5157f0-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.600501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.569342 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.600501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.569377 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.600501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.569493 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.600501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.569560 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.600501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.569586 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7c8d\" (UniqueName: \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-kube-api-access-s7c8d\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.603905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.603861 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.607668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.607632 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.616117 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.616043 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2604e553-ae64-4fdf-ba10-1a639e5157f0-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.616897 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.616873 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:22.620848 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:22.620814 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:23.127155 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.127117 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7c8d\" (UniqueName: \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-kube-api-access-s7c8d\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:23.147663 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.147628 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:23.521233 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.189571 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 13:15:23.546622 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.546580 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 13:15:23.593574 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.212985 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:23.593837 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.213981 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:15:23.595134 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.278805 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:23.628137 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.279356 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:23.648184 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.628724 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:23.722569 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.281006 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:23.753535 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.371214 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:15:23.753535 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.371319 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:23.753535 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.190108 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:23.753535 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.723309 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:23.778560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.776964 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:23.795532 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:23.371076 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:24.756502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.749291 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xtnsg" podUID="fda89da9-cda2-4148-ae7b-70f8966b3769" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Oct 31 13:15:24.758496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.758440 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:24.806522 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.801628 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 13:15:24.879065 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:24.807588 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:15:24.879065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.808743 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:24.879065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.836213 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:15:24.879065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.836301 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 13:15:24.879065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.836345 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 13:15:24.879065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.836574 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg" podUID="8269080b-1265-480e-a205-976decbd5bfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:24.879065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.853444 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:15:24.928511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.925736 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" Oct 31 13:15:24.961575 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.953636 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-c6vhc" podUID="31c91a14-a352-471c-9857-d483ac2c758a" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.23:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:24.961575 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.959697 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:24.963886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.962019 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:15:24.963886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.962342 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 13:15:24.963886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:24.962609 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:15:25.054726 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.043362 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-maas-api-service-ca\") pod \"98a79ba0-5690-4754-8ad0-8689bcaa1ae8\" (UID: \"98a79ba0-5690-4754-8ad0-8689bcaa1ae8\") " Oct 31 13:15:25.185703 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.183919 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-maas-api-service-ca" pod "98a79ba0-5690-4754-8ad0-8689bcaa1ae8" (UID: "98a79ba0-5690-4754-8ad0-8689bcaa1ae8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:25.336498 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:25.322177 4542 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.023s" Oct 31 13:15:25.374573 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.366704 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-kube-api-access-cfw48\" (UniqueName: \"kubernetes.io/projected/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-kube-api-access-cfw48\") pod \"98a79ba0-5690-4754-8ad0-8689bcaa1ae8\" (UID: \"98a79ba0-5690-4754-8ad0-8689bcaa1ae8\") " Oct 31 13:15:25.393081 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.382828 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-maas-api-service-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.444069 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.444033 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-kube-api-access-cfw48" pod "98a79ba0-5690-4754-8ad0-8689bcaa1ae8" (UID: "98a79ba0-5690-4754-8ad0-8689bcaa1ae8"). InnerVolumeSpecName "kube-api-access-cfw48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:25.474537 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.462540 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" Oct 31 13:15:25.804268 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.804175 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfw48\" (UniqueName: \"kubernetes.io/projected/98a79ba0-5690-4754-8ad0-8689bcaa1ae8-kube-api-access-cfw48\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.844803 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.811463 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:25.894249 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.894207 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 31 13:15:25.919247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.913960 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 13:15:25.940877 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.940835 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:25.947247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.941140 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:25.959513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.941192 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:25.961004 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.941215 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:25.961667 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.942351 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecdecd31-e480-4f1d-86c4-a023cc2e418e-kube-api-access-rtknj\" (UniqueName: \"kubernetes.io/projected/ecdecd31-e480-4f1d-86c4-a023cc2e418e-kube-api-access-rtknj\") pod \"ecdecd31-e480-4f1d-86c4-a023cc2e418e\" (UID: \"ecdecd31-e480-4f1d-86c4-a023cc2e418e\") " Oct 31 13:15:25.981034 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.980990 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecdecd31-e480-4f1d-86c4-a023cc2e418e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecdecd31-e480-4f1d-86c4-a023cc2e418e-maas-api-service-ca\") pod \"ecdecd31-e480-4f1d-86c4-a023cc2e418e\" (UID: \"ecdecd31-e480-4f1d-86c4-a023cc2e418e\") " Oct 31 13:15:25.982365 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.982302 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:25.984399 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.984372 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 13:15:25.985713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.985688 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 13:15:26.062312 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:25.990020 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 13:15:26.097169 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.097116 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:26.249489 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.097438 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 31 13:15:26.249758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.098567 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Oct 31 13:15:26.249923 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.156995 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" containerID="cri-o://dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff" gracePeriod=45 Oct 31 13:15:26.250152 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.227161 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" Oct 31 13:15:26.250927 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.249119 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdecd31-e480-4f1d-86c4-a023cc2e418e-kube-api-access-rtknj" pod "ecdecd31-e480-4f1d-86c4-a023cc2e418e" (UID: "ecdecd31-e480-4f1d-86c4-a023cc2e418e"). InnerVolumeSpecName "kube-api-access-rtknj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:26.250927 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.249343 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdecd31-e480-4f1d-86c4-a023cc2e418e-maas-api-service-ca" pod "ecdecd31-e480-4f1d-86c4-a023cc2e418e" (UID: "ecdecd31-e480-4f1d-86c4-a023cc2e418e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.283210 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn"] Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.283297 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.283342 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.283362 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng" event={"ID":"98a79ba0-5690-4754-8ad0-8689bcaa1ae8","Type":"ContainerDied","Data":"4db5a8909751f68c6e09fa1f1c81c30988609a9705d331cd2b111f137b4c83d6"} Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.283423 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.283460 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.283512 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.283549 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.283584 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.300337 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.300417 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.300457 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.300508 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.300528 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw"] Oct 31 13:15:26.363077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.303319 4542 scope.go:122] "RemoveContainer" containerID="1e6cb818324e8a8600a83260cc9d730bd8bb7237f05d0337b810e178ca995fe4" Oct 31 13:15:26.558260 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.558212 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc"] Oct 31 13:15:26.579550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.578255 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:15:26.582662 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.582639 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:15:26.596399 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.596367 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" event={"ID":"ecdecd31-e480-4f1d-86c4-a023cc2e418e","Type":"ContainerDied","Data":"17a0e8563438ca03f56b28f7c6a92624cc2bcafc4052719fb803e3fae6aef478"} Oct 31 13:15:26.743588 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.741711 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtknj\" (UniqueName: \"kubernetes.io/projected/ecdecd31-e480-4f1d-86c4-a023cc2e418e-kube-api-access-rtknj\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:26.743588 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.741810 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecdecd31-e480-4f1d-86c4-a023cc2e418e-maas-api-service-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:26.751810 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.751650 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr" Oct 31 13:15:26.767868 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.767846 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:15:26.771029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.771009 4542 scope.go:122] "RemoveContainer" containerID="a6dc479ffeb71d500fb549e60bc1cc0704d23583e929687ded417cacf9f73ad5" Oct 31 13:15:26.773070 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.773050 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:15:26.775701 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.774014 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": EOF" Oct 31 13:15:26.778216 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.778194 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:15:26.781985 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.781961 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:15:26.784036 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.784008 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": dial tcp 10.133.0.42:8000: connect: connection refused" Oct 31 13:15:26.790619 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.787433 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:15:26.790619 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.788823 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:15:26.800124 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.794469 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:15:26.818490 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.815103 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:15:26.966262 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.966234 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs"] Oct 31 13:15:26.967375 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:15:26.967350 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2604e553_ae64_4fdf_ba10_1a639e5157f0.slice/crio-d9b45ca21d91c64d31c4c5b00939312ea4c5f31f5c96c34d0a5fefd62bec47b1 WatchSource:0}: Error finding container d9b45ca21d91c64d31c4c5b00939312ea4c5f31f5c96c34d0a5fefd62bec47b1: Status 404 returned error can't find the container with id d9b45ca21d91c64d31c4c5b00939312ea4c5f31f5c96c34d0a5fefd62bec47b1 Oct 31 13:15:26.970207 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.970171 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:15:26.970311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.970251 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:15:26.970311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:26.970295 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:15:27.277991 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.277959 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:15:27.354254 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.354219 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr"] Oct 31 13:15:27.361006 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.360978 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f9e24f-29890875-gtbwr"] Oct 31 13:15:27.373127 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.373104 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng"] Oct 31 13:15:27.378827 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.378799 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-004f64-29890875-626ng"] Oct 31 13:15:27.457892 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.457865 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-dshm\") pod \"01c6f328-8a73-413c-a1a8-cc278e20c6db\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " Oct 31 13:15:27.458076 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.457936 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c6f328-8a73-413c-a1a8-cc278e20c6db-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c6f328-8a73-413c-a1a8-cc278e20c6db-tls-certs\") pod \"01c6f328-8a73-413c-a1a8-cc278e20c6db\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " Oct 31 13:15:27.458076 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.457985 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-home\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-home\") pod \"01c6f328-8a73-413c-a1a8-cc278e20c6db\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " Oct 31 13:15:27.458076 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.458021 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-tmp-dir\") pod \"01c6f328-8a73-413c-a1a8-cc278e20c6db\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " Oct 31 13:15:27.458237 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.458133 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-model-cache\") pod \"01c6f328-8a73-413c-a1a8-cc278e20c6db\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " Oct 31 13:15:27.458289 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.458253 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c6f328-8a73-413c-a1a8-cc278e20c6db-kube-api-access-d4kqm\" (UniqueName: \"kubernetes.io/projected/01c6f328-8a73-413c-a1a8-cc278e20c6db-kube-api-access-d4kqm\") pod \"01c6f328-8a73-413c-a1a8-cc278e20c6db\" (UID: \"01c6f328-8a73-413c-a1a8-cc278e20c6db\") " Oct 31 13:15:27.458965 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.458601 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-tmp-dir" pod "01c6f328-8a73-413c-a1a8-cc278e20c6db" (UID: "01c6f328-8a73-413c-a1a8-cc278e20c6db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:27.458965 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.458728 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-model-cache" pod "01c6f328-8a73-413c-a1a8-cc278e20c6db" (UID: "01c6f328-8a73-413c-a1a8-cc278e20c6db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:27.459160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.459133 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.459222 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.459157 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.459983 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.459961 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-home" pod "01c6f328-8a73-413c-a1a8-cc278e20c6db" (UID: "01c6f328-8a73-413c-a1a8-cc278e20c6db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:27.473591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.473565 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c6f328-8a73-413c-a1a8-cc278e20c6db-kube-api-access-d4kqm" pod "01c6f328-8a73-413c-a1a8-cc278e20c6db" (UID: "01c6f328-8a73-413c-a1a8-cc278e20c6db"). InnerVolumeSpecName "kube-api-access-d4kqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:27.473841 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.473813 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-dshm" pod "01c6f328-8a73-413c-a1a8-cc278e20c6db" (UID: "01c6f328-8a73-413c-a1a8-cc278e20c6db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:27.474016 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.473998 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c6f328-8a73-413c-a1a8-cc278e20c6db-tls-certs" pod "01c6f328-8a73-413c-a1a8-cc278e20c6db" (UID: "01c6f328-8a73-413c-a1a8-cc278e20c6db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:27.561186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.561101 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4kqm\" (UniqueName: \"kubernetes.io/projected/01c6f328-8a73-413c-a1a8-cc278e20c6db-kube-api-access-d4kqm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.561186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.561151 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.561186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.561167 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c6f328-8a73-413c-a1a8-cc278e20c6db-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.561186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.561179 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c6f328-8a73-413c-a1a8-cc278e20c6db-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.606410 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.606376 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" event={"ID":"2604e553-ae64-4fdf-ba10-1a639e5157f0","Type":"ContainerStarted","Data":"b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86"} Oct 31 13:15:27.606602 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.606440 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" event={"ID":"2604e553-ae64-4fdf-ba10-1a639e5157f0","Type":"ContainerStarted","Data":"d9b45ca21d91c64d31c4c5b00939312ea4c5f31f5c96c34d0a5fefd62bec47b1"} Oct 31 13:15:27.609138 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.609108 4542 generic.go:356] "Generic (PLEG): container finished" podID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerID="dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff" exitCode=0 Oct 31 13:15:27.609286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.609154 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" Oct 31 13:15:27.609286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.609205 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" event={"ID":"01c6f328-8a73-413c-a1a8-cc278e20c6db","Type":"ContainerDied","Data":"dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff"} Oct 31 13:15:27.609286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.609244 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" event={"ID":"01c6f328-8a73-413c-a1a8-cc278e20c6db","Type":"ContainerDied","Data":"851cf1586ed865bc93aaab7f9cad3619408d7a996634b7c2c41cda1e23c5c5da"} Oct 31 13:15:27.609286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.609266 4542 scope.go:122] "RemoveContainer" containerID="dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff" Oct 31 13:15:27.622283 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.622263 4542 scope.go:122] "RemoveContainer" containerID="dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff" Oct 31 13:15:27.622604 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:27.622585 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff\": container with ID starting with dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff not found: ID does not exist" containerID="dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff" Oct 31 13:15:27.622678 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.622613 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff"} err="failed to get container status \"dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff\": rpc error: code = NotFound desc = could not find container \"dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff\": container with ID starting with dfddb1b576205a2fdce862c1ec4bc6b5dbedbf71e4462d87e15d27085860fcff not found: ID does not exist" Oct 31 13:15:27.633400 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.633359 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" podStartSLOduration=5.633346922 podStartE2EDuration="5.633346922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:27.630833447 +0000 UTC m=+944.475999865" watchObservedRunningTime="2026-10-31 13:15:27.633346922 +0000 UTC m=+944.478513339" Oct 31 13:15:27.657290 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.657262 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg"] Oct 31 13:15:27.665840 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.665812 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg"] Oct 31 13:15:27.792512 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:27.784547 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:28.270636 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:28.270584 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/simulator-w1-a2c006-kserve-75cdfcbd94-dwcsg" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:28.786193 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:28.786155 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 13:15:28.786464 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:28.786431 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 13:15:29.293858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:29.293823 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" path="/var/lib/kubelet/pods/01c6f328-8a73-413c-a1a8-cc278e20c6db/volumes" Oct 31 13:15:29.294332 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:29.294316 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" path="/var/lib/kubelet/pods/98a79ba0-5690-4754-8ad0-8689bcaa1ae8/volumes" Oct 31 13:15:29.294745 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:29.294731 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" path="/var/lib/kubelet/pods/ecdecd31-e480-4f1d-86c4-a023cc2e418e/volumes" Oct 31 13:15:29.787512 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:29.786272 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 13:15:29.787512 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:29.786332 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 13:15:30.781280 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:30.781242 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 13:15:30.781867 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:30.781833 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 13:15:31.779753 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:31.779722 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 13:15:31.779948 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:31.779784 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 13:15:32.669114 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:32.669076 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6"] Oct 31 13:15:32.689680 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:32.689654 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p"] Oct 31 13:15:32.779585 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:32.779548 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 13:15:32.779764 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:32.779603 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 13:15:33.779360 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:33.779324 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 13:15:33.779832 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:33.779380 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 13:15:34.785321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:34.785278 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:34.786608 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:34.786583 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:34.787610 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:34.787583 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:15:38.167146 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:38.167110 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx"] Oct 31 13:15:39.551938 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:39.551900 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t"] Oct 31 13:15:39.555502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:39.552232 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerName="istio-proxy" containerID="cri-o://e2880caaec7243ba4dc57ce5d35c28baf70bf20b1d5573aa8cc2f26eb9d1ddd3" gracePeriod=30 Oct 31 13:15:39.557907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:39.557873 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh"] Oct 31 13:15:39.558309 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:39.558272 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" containerID="cri-o://6490e9e5fb0ed1838508baadbfc058e6bab65c52b4717ac54083c77634d387b5" gracePeriod=30 Oct 31 13:15:39.609191 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:39.609152 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts"] Oct 31 13:15:39.609508 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:39.609452 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" containerID="cri-o://d7ad38a356eb899a6d1058d464691a536736f97180208dcbce66b71c6079e923" gracePeriod=30 Oct 31 13:15:41.760324 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:41.760277 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" containerID="cri-o://42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a" gracePeriod=45 Oct 31 13:15:41.841401 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:41.841362 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused" Oct 31 13:15:41.851319 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:41.851263 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" containerID="cri-o://9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650" gracePeriod=45 Oct 31 13:15:42.198776 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.198748 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:15:42.225556 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.225524 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:15:42.320132 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.320107 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-dshm\") pod \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " Oct 31 13:15:42.320328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.320314 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tmp-dir\") pod \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " Oct 31 13:15:42.320471 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.320460 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78fec81e-a3da-4857-9aa8-555bd89744fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/78fec81e-a3da-4857-9aa8-555bd89744fc-tls-certs\") pod \"78fec81e-a3da-4857-9aa8-555bd89744fc\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " Oct 31 13:15:42.320711 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.320697 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-dshm\") pod \"78fec81e-a3da-4857-9aa8-555bd89744fc\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " Oct 31 13:15:42.320857 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.320847 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-tmp-dir\") pod \"78fec81e-a3da-4857-9aa8-555bd89744fc\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " Oct 31 13:15:42.320929 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.320921 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-home\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-home\") pod \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " Oct 31 13:15:42.321031 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.321022 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-model-cache\") pod \"78fec81e-a3da-4857-9aa8-555bd89744fc\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " Oct 31 13:15:42.321121 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.321111 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tls-certs\") pod \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " Oct 31 13:15:42.321223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.321202 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-model-cache\") pod \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " Oct 31 13:15:42.321361 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.321344 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-home\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-home\") pod \"78fec81e-a3da-4857-9aa8-555bd89744fc\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " Oct 31 13:15:42.322240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.322218 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d065383-afaa-4679-9d3e-3da3ad6ddba6-kube-api-access-8pn6f\" (UniqueName: \"kubernetes.io/projected/0d065383-afaa-4679-9d3e-3da3ad6ddba6-kube-api-access-8pn6f\") pod \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\" (UID: \"0d065383-afaa-4679-9d3e-3da3ad6ddba6\") " Oct 31 13:15:42.322409 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.322392 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78fec81e-a3da-4857-9aa8-555bd89744fc-kube-api-access-wqmhj\" (UniqueName: \"kubernetes.io/projected/78fec81e-a3da-4857-9aa8-555bd89744fc-kube-api-access-wqmhj\") pod \"78fec81e-a3da-4857-9aa8-555bd89744fc\" (UID: \"78fec81e-a3da-4857-9aa8-555bd89744fc\") " Oct 31 13:15:42.323916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.321444 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tmp-dir" pod "0d065383-afaa-4679-9d3e-3da3ad6ddba6" (UID: "0d065383-afaa-4679-9d3e-3da3ad6ddba6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:42.323916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.321675 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-home" pod "0d065383-afaa-4679-9d3e-3da3ad6ddba6" (UID: "0d065383-afaa-4679-9d3e-3da3ad6ddba6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:42.323916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.321843 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-home" pod "78fec81e-a3da-4857-9aa8-555bd89744fc" (UID: "78fec81e-a3da-4857-9aa8-555bd89744fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:42.323916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.321858 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-tmp-dir" pod "78fec81e-a3da-4857-9aa8-555bd89744fc" (UID: "78fec81e-a3da-4857-9aa8-555bd89744fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:42.323916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.322025 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-model-cache" pod "78fec81e-a3da-4857-9aa8-555bd89744fc" (UID: "78fec81e-a3da-4857-9aa8-555bd89744fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:42.323916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.322133 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-model-cache" pod "0d065383-afaa-4679-9d3e-3da3ad6ddba6" (UID: "0d065383-afaa-4679-9d3e-3da3ad6ddba6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:42.323916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.323325 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-dshm" pod "0d065383-afaa-4679-9d3e-3da3ad6ddba6" (UID: "0d065383-afaa-4679-9d3e-3da3ad6ddba6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:42.324812 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.324744 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78fec81e-a3da-4857-9aa8-555bd89744fc-tls-certs" pod "78fec81e-a3da-4857-9aa8-555bd89744fc" (UID: "78fec81e-a3da-4857-9aa8-555bd89744fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:42.325503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.325465 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fec81e-a3da-4857-9aa8-555bd89744fc-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.325591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.325503 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.325591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.325519 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.325591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.325532 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.325591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.325545 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.325591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.325558 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.325591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.325570 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.325591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.325583 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.326739 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.326712 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d065383-afaa-4679-9d3e-3da3ad6ddba6-kube-api-access-8pn6f" pod "0d065383-afaa-4679-9d3e-3da3ad6ddba6" (UID: "0d065383-afaa-4679-9d3e-3da3ad6ddba6"). InnerVolumeSpecName "kube-api-access-8pn6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:42.326898 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.326876 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-dshm" pod "78fec81e-a3da-4857-9aa8-555bd89744fc" (UID: "78fec81e-a3da-4857-9aa8-555bd89744fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:42.327039 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.327018 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tls-certs" pod "0d065383-afaa-4679-9d3e-3da3ad6ddba6" (UID: "0d065383-afaa-4679-9d3e-3da3ad6ddba6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:42.328829 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.328797 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fec81e-a3da-4857-9aa8-555bd89744fc-kube-api-access-wqmhj" pod "78fec81e-a3da-4857-9aa8-555bd89744fc" (UID: "78fec81e-a3da-4857-9aa8-555bd89744fc"). InnerVolumeSpecName "kube-api-access-wqmhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:42.427250 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.427214 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fec81e-a3da-4857-9aa8-555bd89744fc-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.427250 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.427245 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d065383-afaa-4679-9d3e-3da3ad6ddba6-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.427250 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.427256 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pn6f\" (UniqueName: \"kubernetes.io/projected/0d065383-afaa-4679-9d3e-3da3ad6ddba6-kube-api-access-8pn6f\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.427491 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.427265 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqmhj\" (UniqueName: \"kubernetes.io/projected/78fec81e-a3da-4857-9aa8-555bd89744fc-kube-api-access-wqmhj\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:42.633780 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.633687 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" containerID="cri-o://f13423420aec1bf34dd75dd62961e2df3b2567c0b2b297786aa23b43593475fd" gracePeriod=45 Oct 31 13:15:42.868580 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.856319 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused" Oct 31 13:15:42.892975 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.892940 4542 generic.go:356] "Generic (PLEG): container finished" podID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerID="f13423420aec1bf34dd75dd62961e2df3b2567c0b2b297786aa23b43593475fd" exitCode=0 Oct 31 13:15:42.893167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.893141 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" event={"ID":"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d","Type":"ContainerDied","Data":"f13423420aec1bf34dd75dd62961e2df3b2567c0b2b297786aa23b43593475fd"} Oct 31 13:15:42.894769 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.894739 4542 generic.go:356] "Generic (PLEG): container finished" podID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerID="9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650" exitCode=0 Oct 31 13:15:42.894976 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.894785 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" Oct 31 13:15:42.895372 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.894801 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" event={"ID":"78fec81e-a3da-4857-9aa8-555bd89744fc","Type":"ContainerDied","Data":"9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650"} Oct 31 13:15:42.895461 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.895397 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw" event={"ID":"78fec81e-a3da-4857-9aa8-555bd89744fc","Type":"ContainerDied","Data":"bbb4b0e2eccf70a99a271a02031f6fec18366b0716dd5e45730a3b2b4020c4f8"} Oct 31 13:15:42.895461 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.895424 4542 scope.go:122] "RemoveContainer" containerID="9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650" Oct 31 13:15:42.897729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.897646 4542 generic.go:356] "Generic (PLEG): container finished" podID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerID="42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a" exitCode=0 Oct 31 13:15:42.897729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.897686 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" event={"ID":"0d065383-afaa-4679-9d3e-3da3ad6ddba6","Type":"ContainerDied","Data":"42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a"} Oct 31 13:15:42.897729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.897711 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" Oct 31 13:15:42.898065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.897715 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn" event={"ID":"0d065383-afaa-4679-9d3e-3da3ad6ddba6","Type":"ContainerDied","Data":"669088fc93d783eaacc16ae4927d339c7641f9e18e00be86d295117d4176632b"} Oct 31 13:15:42.928094 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.928063 4542 scope.go:122] "RemoveContainer" containerID="9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650" Oct 31 13:15:42.928718 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:42.928697 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650\": container with ID starting with 9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650 not found: ID does not exist" containerID="9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650" Oct 31 13:15:42.929547 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.928732 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650"} err="failed to get container status \"9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650\": rpc error: code = NotFound desc = could not find container \"9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650\": container with ID starting with 9ebe89803cbad27afe2c9e0817de5248596ed6708f3ad0440e5d51029fb52650 not found: ID does not exist" Oct 31 13:15:42.929547 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.928755 4542 scope.go:122] "RemoveContainer" containerID="42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a" Oct 31 13:15:42.935492 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.935426 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw"] Oct 31 13:15:42.944642 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.944337 4542 scope.go:122] "RemoveContainer" containerID="42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a" Oct 31 13:15:42.944820 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:42.944788 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a\": container with ID starting with 42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a not found: ID does not exist" containerID="42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a" Oct 31 13:15:42.944996 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.944964 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a"} err="failed to get container status \"42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a\": rpc error: code = NotFound desc = could not find container \"42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a\": container with ID starting with 42bd25c839dafcac0204f30deb2489180e778c19459210865f6b5d1b9418b79a not found: ID does not exist" Oct 31 13:15:42.947642 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.947613 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-w2-2a262a-kserve-6f757595d7-x5jpw"] Oct 31 13:15:42.962316 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.962293 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn"] Oct 31 13:15:42.976069 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:42.976042 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/distinct-2-w2-2a262a-kserve-7c57cf6876-t6dmn"] Oct 31 13:15:43.080352 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.080325 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:15:43.151543 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.151423 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tls-certs\") pod \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " Oct 31 13:15:43.151543 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.151512 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tmp-dir\") pod \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " Oct 31 13:15:43.151749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.151652 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-model-cache\") pod \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " Oct 31 13:15:43.151749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.151722 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-kube-api-access-tk2lg\" (UniqueName: \"kubernetes.io/projected/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-kube-api-access-tk2lg\") pod \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " Oct 31 13:15:43.151873 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.151758 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-home\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-home\") pod \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " Oct 31 13:15:43.151929 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.151878 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-dshm\") pod \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\" (UID: \"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d\") " Oct 31 13:15:43.152160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.152112 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-model-cache" pod "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" (UID: "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:43.152923 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.152270 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tmp-dir" pod "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" (UID: "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:43.152923 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.152707 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-home" pod "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" (UID: "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:43.153314 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.153284 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:43.153314 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.153314 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:43.153522 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.153329 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:43.154187 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.154164 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tls-certs" pod "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" (UID: "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:43.155037 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.155011 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-kube-api-access-tk2lg" pod "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" (UID: "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d"). InnerVolumeSpecName "kube-api-access-tk2lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:43.155368 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.155334 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-dshm" pod "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" (UID: "7af893f2-4ce1-4c3b-8d4b-05b17617ab5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:43.168463 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.168435 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" podUID="c3f742af-293a-40d6-9a93-75e3030306b8" containerName="payload-pre-processing" containerID="cri-o://67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6" gracePeriod=25 Oct 31 13:15:43.256637 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.256608 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:43.256637 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.256641 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:43.256914 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.256656 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2lg\" (UniqueName: \"kubernetes.io/projected/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d-kube-api-access-tk2lg\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:43.294545 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.294510 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" path="/var/lib/kubelet/pods/0d065383-afaa-4679-9d3e-3da3ad6ddba6/volumes" Oct 31 13:15:43.294996 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.294981 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" path="/var/lib/kubelet/pods/78fec81e-a3da-4857-9aa8-555bd89744fc/volumes" Oct 31 13:15:43.359623 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.359589 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:15:43.359774 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.359648 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:15:43.844523 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.844488 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn"] Oct 31 13:15:43.844750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.844721 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" podUID="d3a1293a-a69b-4a71-95e2-265a03fd633b" containerName="maas-api" containerID="cri-o://6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073" gracePeriod=30 Oct 31 13:15:43.904874 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.904850 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" Oct 31 13:15:43.905274 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.904883 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc" event={"ID":"7af893f2-4ce1-4c3b-8d4b-05b17617ab5d","Type":"ContainerDied","Data":"0b785328b817ec5ebb291937082239bfe635fc4461c43c48f99194c2b742c9f2"} Oct 31 13:15:43.905274 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.904939 4542 scope.go:122] "RemoveContainer" containerID="f13423420aec1bf34dd75dd62961e2df3b2567c0b2b297786aa23b43593475fd" Oct 31 13:15:43.916995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.916970 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" podUID="d3a1293a-a69b-4a71-95e2-265a03fd633b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.38:8443/health\": dial tcp 10.133.0.38:8443: connect: connection refused" Oct 31 13:15:43.936695 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.936673 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc"] Oct 31 13:15:43.940944 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:43.940923 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/premium-w2-2a262a-kserve-99997b4d-8cpvc"] Oct 31 13:15:44.211877 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.211844 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:15:44.265356 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.265318 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" podUID="c3f742af-293a-40d6-9a93-75e3030306b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:15:44.273715 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.273686 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-maas-api-tls\") pod \"d3a1293a-a69b-4a71-95e2-265a03fd633b\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " Oct 31 13:15:44.273895 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.273820 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-metrics-certs\") pod \"d3a1293a-a69b-4a71-95e2-265a03fd633b\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " Oct 31 13:15:44.273959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.273892 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a1293a-a69b-4a71-95e2-265a03fd633b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a1293a-a69b-4a71-95e2-265a03fd633b-openshift-service-ca\") pod \"d3a1293a-a69b-4a71-95e2-265a03fd633b\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " Oct 31 13:15:44.273959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.273940 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a1293a-a69b-4a71-95e2-265a03fd633b-kube-api-access-7bl59\" (UniqueName: \"kubernetes.io/projected/d3a1293a-a69b-4a71-95e2-265a03fd633b-kube-api-access-7bl59\") pod \"d3a1293a-a69b-4a71-95e2-265a03fd633b\" (UID: \"d3a1293a-a69b-4a71-95e2-265a03fd633b\") " Oct 31 13:15:44.275426 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.275399 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a1293a-a69b-4a71-95e2-265a03fd633b-openshift-service-ca" pod "d3a1293a-a69b-4a71-95e2-265a03fd633b" (UID: "d3a1293a-a69b-4a71-95e2-265a03fd633b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:44.282520 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.277946 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a1293a-a69b-4a71-95e2-265a03fd633b-kube-api-access-7bl59" pod "d3a1293a-a69b-4a71-95e2-265a03fd633b" (UID: "d3a1293a-a69b-4a71-95e2-265a03fd633b"). InnerVolumeSpecName "kube-api-access-7bl59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:44.284804 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.284783 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-maas-api-tls" pod "d3a1293a-a69b-4a71-95e2-265a03fd633b" (UID: "d3a1293a-a69b-4a71-95e2-265a03fd633b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:44.286159 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.286135 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-metrics-certs" pod "d3a1293a-a69b-4a71-95e2-265a03fd633b" (UID: "d3a1293a-a69b-4a71-95e2-265a03fd633b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:44.376797 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.376760 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-maas-api-tls\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:44.376797 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.376794 4542 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a1293a-a69b-4a71-95e2-265a03fd633b-metrics-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:44.376797 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.376807 4542 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a1293a-a69b-4a71-95e2-265a03fd633b-openshift-service-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:44.377170 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.376817 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bl59\" (UniqueName: \"kubernetes.io/projected/d3a1293a-a69b-4a71-95e2-265a03fd633b-kube-api-access-7bl59\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:44.623608 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:44.623551 4542 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de77bc5_25f8_4e11_ba04_f3f254a594fc.slice/crio-conmon-e2880caaec7243ba4dc57ce5d35c28baf70bf20b1d5573aa8cc2f26eb9d1ddd3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:15:44.878156 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.877532 4542 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:15:44.878156 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.877585 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.887561 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l"] Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888516 4542 cpu_manager.go:414] "containerMap: removing container" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888538 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888560 4542 cpu_manager.go:414] "containerMap: removing container" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888568 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888581 4542 cpu_manager.go:414] "containerMap: removing container" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888588 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888601 4542 cpu_manager.go:414] "containerMap: removing container" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888608 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888641 4542 cpu_manager.go:414] "containerMap: removing container" podUID="d3a1293a-a69b-4a71-95e2-265a03fd633b" containerName="maas-api" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888649 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a1293a-a69b-4a71-95e2-265a03fd633b" containerName="maas-api" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888667 4542 cpu_manager.go:414] "containerMap: removing container" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888675 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888687 4542 cpu_manager.go:414] "containerMap: removing container" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888695 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888720 4542 cpu_manager.go:414] "containerMap: removing container" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888726 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888734 4542 cpu_manager.go:414] "containerMap: removing container" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888741 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888866 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888878 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888888 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="78fec81e-a3da-4857-9aa8-555bd89744fc" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888906 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a1293a-a69b-4a71-95e2-265a03fd633b" containerName="maas-api" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888917 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888926 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c6f328-8a73-413c-a1a8-cc278e20c6db" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888938 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerName="cleanup" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888948 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d065383-afaa-4679-9d3e-3da3ad6ddba6" containerName="main" Oct 31 13:15:44.894015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.888960 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" containerName="main" Oct 31 13:15:44.921871 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.921832 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:44.931280 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.929426 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:15:44.931280 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.929607 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\"" Oct 31 13:15:44.945202 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.945167 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l"] Oct 31 13:15:44.945768 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.945732 4542 generic.go:356] "Generic (PLEG): container finished" podID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerID="d7ad38a356eb899a6d1058d464691a536736f97180208dcbce66b71c6079e923" exitCode=0 Oct 31 13:15:44.946087 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.946061 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" event={"ID":"b1cb27e2-2458-4c4f-8b00-b145c9bd8116","Type":"ContainerDied","Data":"d7ad38a356eb899a6d1058d464691a536736f97180208dcbce66b71c6079e923"} Oct 31 13:15:44.952167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.952138 4542 generic.go:356] "Generic (PLEG): container finished" podID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerID="6490e9e5fb0ed1838508baadbfc058e6bab65c52b4717ac54083c77634d387b5" exitCode=0 Oct 31 13:15:44.952416 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.952392 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" event={"ID":"ab0d0c59-d655-4733-993c-d89bca07faf9","Type":"ContainerDied","Data":"6490e9e5fb0ed1838508baadbfc058e6bab65c52b4717ac54083c77634d387b5"} Oct 31 13:15:44.952602 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.952583 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" event={"ID":"ab0d0c59-d655-4733-993c-d89bca07faf9","Type":"ContainerDied","Data":"7edbf413a75e833af0415f7f146bfe013c2f70fae4576dd0ca284088439e4335"} Oct 31 13:15:44.952738 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.952722 4542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7edbf413a75e833af0415f7f146bfe013c2f70fae4576dd0ca284088439e4335" Oct 31 13:15:44.952850 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.952837 4542 scope.go:122] "RemoveContainer" containerID="6844ab034a05f4a8df5cb0d0595f8fd00bbd52c415fe6ad001d618f185e7a75f" Oct 31 13:15:44.955400 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.955378 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:15:44.983753 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.979404 4542 generic.go:356] "Generic (PLEG): container finished" podID="d3a1293a-a69b-4a71-95e2-265a03fd633b" containerID="6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073" exitCode=0 Oct 31 13:15:44.983753 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.979572 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" Oct 31 13:15:44.983753 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.981444 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" event={"ID":"d3a1293a-a69b-4a71-95e2-265a03fd633b","Type":"ContainerDied","Data":"6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073"} Oct 31 13:15:44.983753 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.981554 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn" event={"ID":"d3a1293a-a69b-4a71-95e2-265a03fd633b","Type":"ContainerDied","Data":"3a5c9d7b3462a86fc4e39b02ebdafb6533b726842cb562ebaf615f9d64ad71c7"} Oct 31 13:15:44.988952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.988917 4542 generic.go:356] "Generic (PLEG): container finished" podID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerID="e2880caaec7243ba4dc57ce5d35c28baf70bf20b1d5573aa8cc2f26eb9d1ddd3" exitCode=0 Oct 31 13:15:44.989115 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.989017 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" event={"ID":"2de77bc5-25f8-4e11-ba04-f3f254a594fc","Type":"ContainerDied","Data":"e2880caaec7243ba4dc57ce5d35c28baf70bf20b1d5573aa8cc2f26eb9d1ddd3"} Oct 31 13:15:44.990569 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:44.990550 4542 scope.go:122] "RemoveContainer" containerID="6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073" Oct 31 13:15:45.018919 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.018882 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zq8w\" (UniqueName: \"kubernetes.io/projected/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-kube-api-access-2zq8w\") pod \"maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:45.019042 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.019012 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:45.019168 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.019148 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:45.019409 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.019389 4542 scope.go:122] "RemoveContainer" containerID="6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073" Oct 31 13:15:45.026965 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:45.026930 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073\": container with ID starting with 6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073 not found: ID does not exist" containerID="6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073" Oct 31 13:15:45.027057 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.026978 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073"} err="failed to get container status \"6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073\": rpc error: code = NotFound desc = could not find container \"6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073\": container with ID starting with 6a158d4b12b5655ef70f186ef88a1d5f164fe6879320e3807fc81454c79a4073 not found: ID does not exist" Oct 31 13:15:45.045986 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.045951 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn"] Oct 31 13:15:45.067657 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.067612 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a2c006-7bdfcbd65d-lw4bn"] Oct 31 13:15:45.151968 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.151925 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-envoy\") pod \"ab0d0c59-d655-4733-993c-d89bca07faf9\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " Oct 31 13:15:45.152192 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.152009 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0d0c59-d655-4733-993c-d89bca07faf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0d0c59-d655-4733-993c-d89bca07faf9-istiod-ca-cert\") pod \"ab0d0c59-d655-4733-993c-d89bca07faf9\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " Oct 31 13:15:45.152192 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.152106 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-certs\") pod \"ab0d0c59-d655-4733-993c-d89bca07faf9\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " Oct 31 13:15:45.152192 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.152145 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-data\") pod \"ab0d0c59-d655-4733-993c-d89bca07faf9\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " Oct 31 13:15:45.152192 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.152173 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-kube-api-access-jxpbs\" (UniqueName: \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-kube-api-access-jxpbs\") pod \"ab0d0c59-d655-4733-993c-d89bca07faf9\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " Oct 31 13:15:45.152671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.152257 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-socket\") pod \"ab0d0c59-d655-4733-993c-d89bca07faf9\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " Oct 31 13:15:45.152671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.152343 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-podinfo\") pod \"ab0d0c59-d655-4733-993c-d89bca07faf9\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " Oct 31 13:15:45.152671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.152412 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-token\" (UniqueName: \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-token\") pod \"ab0d0c59-d655-4733-993c-d89bca07faf9\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " Oct 31 13:15:45.152671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.152425 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-certs" pod "ab0d0c59-d655-4733-993c-d89bca07faf9" (UID: "ab0d0c59-d655-4733-993c-d89bca07faf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.152671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.152559 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-credential-socket\") pod \"ab0d0c59-d655-4733-993c-d89bca07faf9\" (UID: \"ab0d0c59-d655-4733-993c-d89bca07faf9\") " Oct 31 13:15:45.153368 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.153339 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-data" pod "ab0d0c59-d655-4733-993c-d89bca07faf9" (UID: "ab0d0c59-d655-4733-993c-d89bca07faf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.153544 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.153405 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-socket" pod "ab0d0c59-d655-4733-993c-d89bca07faf9" (UID: "ab0d0c59-d655-4733-993c-d89bca07faf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.153622 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.153607 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-credential-socket" pod "ab0d0c59-d655-4733-993c-d89bca07faf9" (UID: "ab0d0c59-d655-4733-993c-d89bca07faf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.154054 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.154032 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0d0c59-d655-4733-993c-d89bca07faf9-istiod-ca-cert" pod "ab0d0c59-d655-4733-993c-d89bca07faf9" (UID: "ab0d0c59-d655-4733-993c-d89bca07faf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:45.161872 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.160903 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-podinfo" pod "ab0d0c59-d655-4733-993c-d89bca07faf9" (UID: "ab0d0c59-d655-4733-993c-d89bca07faf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:15:45.162856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.162831 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zq8w\" (UniqueName: \"kubernetes.io/projected/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-kube-api-access-2zq8w\") pod \"maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:45.163699 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.163669 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:45.163816 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.163145 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-kube-api-access-jxpbs" pod "ab0d0c59-d655-4733-993c-d89bca07faf9" (UID: "ab0d0c59-d655-4733-993c-d89bca07faf9"). InnerVolumeSpecName "kube-api-access-jxpbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:45.163878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.163866 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:45.164006 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.163990 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.164077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.164013 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0d0c59-d655-4733-993c-d89bca07faf9-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.164077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.164023 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.164077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.164031 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.164077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.164039 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxpbs\" (UniqueName: \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-kube-api-access-jxpbs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.164077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.164052 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.164077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.164064 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.164624 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.164599 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:45.164826 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.163350 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-token" pod "ab0d0c59-d655-4733-993c-d89bca07faf9" (UID: "ab0d0c59-d655-4733-993c-d89bca07faf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:45.164994 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.164972 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-envoy" pod "ab0d0c59-d655-4733-993c-d89bca07faf9" (UID: "ab0d0c59-d655-4733-993c-d89bca07faf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.167065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.167045 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:45.178031 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.178007 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:15:45.179267 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.179240 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zq8w\" (UniqueName: \"kubernetes.io/projected/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-kube-api-access-2zq8w\") pod \"maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:45.181870 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.181851 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:15:45.266299 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266261 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istiod-ca-cert\") pod \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " Oct 31 13:15:45.266607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266343 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-certs\") pod \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " Oct 31 13:15:45.266607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266374 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-envoy\") pod \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " Oct 31 13:15:45.266607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266418 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-podinfo\") pod \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " Oct 31 13:15:45.266607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266525 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-certs\") pod \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " Oct 31 13:15:45.266607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266584 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-token\") pod \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " Oct 31 13:15:45.267014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266659 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istiod-ca-cert\") pod \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " Oct 31 13:15:45.267014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266732 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-socket\") pod \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " Oct 31 13:15:45.267014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266781 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-credential-socket\") pod \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " Oct 31 13:15:45.267014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266822 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-credential-socket\") pod \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " Oct 31 13:15:45.267014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266852 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-data\") pod \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " Oct 31 13:15:45.267014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266869 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istiod-ca-cert" pod "2de77bc5-25f8-4e11-ba04-f3f254a594fc" (UID: "2de77bc5-25f8-4e11-ba04-f3f254a594fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:45.267014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266924 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-kube-api-access-r5jdj\" (UniqueName: \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-kube-api-access-r5jdj\") pod \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " Oct 31 13:15:45.267014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266954 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-kube-api-access-7q9z9\" (UniqueName: \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-kube-api-access-7q9z9\") pod \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " Oct 31 13:15:45.267014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.266992 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-podinfo\") pod \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " Oct 31 13:15:45.267455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.267113 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-socket\") pod \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " Oct 31 13:15:45.267455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.267186 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-certs" pod "2de77bc5-25f8-4e11-ba04-f3f254a594fc" (UID: "2de77bc5-25f8-4e11-ba04-f3f254a594fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.267455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.267203 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-token\" (UniqueName: \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-token\") pod \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " Oct 31 13:15:45.267455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.267249 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-data\") pod \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\" (UID: \"b1cb27e2-2458-4c4f-8b00-b145c9bd8116\") " Oct 31 13:15:45.267455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.267265 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istiod-ca-cert" pod "b1cb27e2-2458-4c4f-8b00-b145c9bd8116" (UID: "b1cb27e2-2458-4c4f-8b00-b145c9bd8116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:45.267455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.267320 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-envoy\") pod \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\" (UID: \"2de77bc5-25f8-4e11-ba04-f3f254a594fc\") " Oct 31 13:15:45.267455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.267399 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-data" pod "2de77bc5-25f8-4e11-ba04-f3f254a594fc" (UID: "2de77bc5-25f8-4e11-ba04-f3f254a594fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.267805 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.267647 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-socket" pod "2de77bc5-25f8-4e11-ba04-f3f254a594fc" (UID: "2de77bc5-25f8-4e11-ba04-f3f254a594fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.268180 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.268134 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.268180 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.268167 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.268594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.268183 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.268594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.268197 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0d0c59-d655-4733-993c-d89bca07faf9-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.268594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.268212 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.268594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.268226 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.268594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.268240 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.268594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.268182 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-credential-socket" pod "2de77bc5-25f8-4e11-ba04-f3f254a594fc" (UID: "2de77bc5-25f8-4e11-ba04-f3f254a594fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.268594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.268432 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-credential-socket" pod "b1cb27e2-2458-4c4f-8b00-b145c9bd8116" (UID: "b1cb27e2-2458-4c4f-8b00-b145c9bd8116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.269109 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.268968 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-certs" pod "b1cb27e2-2458-4c4f-8b00-b145c9bd8116" (UID: "b1cb27e2-2458-4c4f-8b00-b145c9bd8116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.269182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.269158 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-data" pod "b1cb27e2-2458-4c4f-8b00-b145c9bd8116" (UID: "b1cb27e2-2458-4c4f-8b00-b145c9bd8116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.269359 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.269338 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-socket" pod "b1cb27e2-2458-4c4f-8b00-b145c9bd8116" (UID: "b1cb27e2-2458-4c4f-8b00-b145c9bd8116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.271103 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.271073 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-kube-api-access-r5jdj" pod "b1cb27e2-2458-4c4f-8b00-b145c9bd8116" (UID: "b1cb27e2-2458-4c4f-8b00-b145c9bd8116"). InnerVolumeSpecName "kube-api-access-r5jdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:45.272617 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.272584 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-kube-api-access-7q9z9" pod "2de77bc5-25f8-4e11-ba04-f3f254a594fc" (UID: "2de77bc5-25f8-4e11-ba04-f3f254a594fc"). InnerVolumeSpecName "kube-api-access-7q9z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:45.277917 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.277884 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-token" pod "2de77bc5-25f8-4e11-ba04-f3f254a594fc" (UID: "2de77bc5-25f8-4e11-ba04-f3f254a594fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:45.279218 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.279187 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-token" pod "b1cb27e2-2458-4c4f-8b00-b145c9bd8116" (UID: "b1cb27e2-2458-4c4f-8b00-b145c9bd8116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:45.280887 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.280744 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-envoy" pod "2de77bc5-25f8-4e11-ba04-f3f254a594fc" (UID: "2de77bc5-25f8-4e11-ba04-f3f254a594fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.280887 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.280832 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-envoy" pod "b1cb27e2-2458-4c4f-8b00-b145c9bd8116" (UID: "b1cb27e2-2458-4c4f-8b00-b145c9bd8116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:45.282144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.282118 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-podinfo" pod "b1cb27e2-2458-4c4f-8b00-b145c9bd8116" (UID: "b1cb27e2-2458-4c4f-8b00-b145c9bd8116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:15:45.282244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.282197 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-podinfo" pod "2de77bc5-25f8-4e11-ba04-f3f254a594fc" (UID: "2de77bc5-25f8-4e11-ba04-f3f254a594fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:15:45.288401 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.288381 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:45.293308 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.293289 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af893f2-4ce1-4c3b-8d4b-05b17617ab5d" path="/var/lib/kubelet/pods/7af893f2-4ce1-4c3b-8d4b-05b17617ab5d/volumes" Oct 31 13:15:45.293845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.293830 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a1293a-a69b-4a71-95e2-265a03fd633b" path="/var/lib/kubelet/pods/d3a1293a-a69b-4a71-95e2-265a03fd633b/volumes" Oct 31 13:15:45.329330 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.329284 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg"] Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.369985 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370014 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370027 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370037 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370047 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5jdj\" (UniqueName: \"kubernetes.io/projected/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-kube-api-access-r5jdj\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370057 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q9z9\" (UniqueName: \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-kube-api-access-7q9z9\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370065 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370074 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370082 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370090 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370099 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370106 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb27e2-2458-4c4f-8b00-b145c9bd8116-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.370158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.370116 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de77bc5-25f8-4e11-ba04-f3f254a594fc-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:45.596226 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:15:45.596196 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bcbb4d_b174_4a1d_b5ce_6e28726bcf74.slice/crio-19d9ee4941dc71699da795b830b1d569b1e3db2620d756740364efa280a48b31 WatchSource:0}: Error finding container 19d9ee4941dc71699da795b830b1d569b1e3db2620d756740364efa280a48b31: Status 404 returned error can't find the container with id 19d9ee4941dc71699da795b830b1d569b1e3db2620d756740364efa280a48b31 Oct 31 13:15:45.596630 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.596598 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l"] Oct 31 13:15:45.998189 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.998164 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" Oct 31 13:15:45.998705 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.998188 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts" event={"ID":"b1cb27e2-2458-4c4f-8b00-b145c9bd8116","Type":"ContainerDied","Data":"f4583b171fb169899d987adcd50a46ef7e143c835a687fd1a513e4c2759d8a4a"} Oct 31 13:15:45.998705 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.998250 4542 scope.go:122] "RemoveContainer" containerID="d7ad38a356eb899a6d1058d464691a536736f97180208dcbce66b71c6079e923" Oct 31 13:15:45.999575 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:45.999555 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh" Oct 31 13:15:46.003276 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.003248 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" Oct 31 13:15:46.003870 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.003841 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t" event={"ID":"2de77bc5-25f8-4e11-ba04-f3f254a594fc","Type":"ContainerDied","Data":"4595479e37229430335b6cb70e85411322a748c99ecfbc450e4a41cf4796ec23"} Oct 31 13:15:46.005749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.005723 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" event={"ID":"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74","Type":"ContainerStarted","Data":"90c18c3f49486cd76005f2cc169a43ddd8640fc1f17c82f74a4fb32c139d679a"} Oct 31 13:15:46.005861 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.005774 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" event={"ID":"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74","Type":"ContainerStarted","Data":"19d9ee4941dc71699da795b830b1d569b1e3db2620d756740364efa280a48b31"} Oct 31 13:15:46.009412 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.009393 4542 scope.go:122] "RemoveContainer" containerID="e2880caaec7243ba4dc57ce5d35c28baf70bf20b1d5573aa8cc2f26eb9d1ddd3" Oct 31 13:15:46.060156 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.060114 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh"] Oct 31 13:15:46.070555 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.066136 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-49gqh"] Oct 31 13:15:46.092579 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.092546 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts"] Oct 31 13:15:46.096202 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.096176 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-8rbts"] Oct 31 13:15:46.111422 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.111348 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t"] Oct 31 13:15:46.118553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.118527 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-sf48t"] Oct 31 13:15:46.147139 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:46.147088 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" podStartSLOduration=2.147070188 podStartE2EDuration="2.147070188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:46.146608402 +0000 UTC m=+962.991774821" watchObservedRunningTime="2026-10-31 13:15:46.147070188 +0000 UTC m=+962.992236606" Oct 31 13:15:47.025123 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:47.025081 4542 generic.go:356] "Generic (PLEG): container finished" podID="46bcbb4d-b174-4a1d-b5ce-6e28726bcf74" containerID="90c18c3f49486cd76005f2cc169a43ddd8640fc1f17c82f74a4fb32c139d679a" exitCode=0 Oct 31 13:15:47.025562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:47.025170 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" event={"ID":"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74","Type":"ContainerDied","Data":"90c18c3f49486cd76005f2cc169a43ddd8640fc1f17c82f74a4fb32c139d679a"} Oct 31 13:15:47.296009 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:47.295935 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" path="/var/lib/kubelet/pods/2de77bc5-25f8-4e11-ba04-f3f254a594fc/volumes" Oct 31 13:15:47.296432 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:47.296418 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" path="/var/lib/kubelet/pods/ab0d0c59-d655-4733-993c-d89bca07faf9/volumes" Oct 31 13:15:47.296996 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:47.296979 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" path="/var/lib/kubelet/pods/b1cb27e2-2458-4c4f-8b00-b145c9bd8116/volumes" Oct 31 13:15:47.692962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:47.692920 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" containerID="cri-o://0191a9bf51a7e7763c4841c141ea03c319fa07e05577e9d48ff8b9a4b9fb23e3" gracePeriod=45 Oct 31 13:15:47.711061 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:47.711024 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" containerID="cri-o://b73524eec74a6cf109978f6d10642882ffa26f38a8be05e8c2113525b3f0c2a9" gracePeriod=45 Oct 31 13:15:47.773660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:47.773624 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": dial tcp 10.133.0.64:8000: connect: connection refused" Oct 31 13:15:47.778036 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:47.778004 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": dial tcp 10.133.0.43:8000: connect: connection refused" Oct 31 13:15:48.037246 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.037207 4542 generic.go:356] "Generic (PLEG): container finished" podID="03559efd-42ab-4720-9111-a230f6c30c41" containerID="0191a9bf51a7e7763c4841c141ea03c319fa07e05577e9d48ff8b9a4b9fb23e3" exitCode=0 Oct 31 13:15:48.037907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.037336 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" event={"ID":"03559efd-42ab-4720-9111-a230f6c30c41","Type":"ContainerDied","Data":"0191a9bf51a7e7763c4841c141ea03c319fa07e05577e9d48ff8b9a4b9fb23e3"} Oct 31 13:15:48.037907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.037374 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" event={"ID":"03559efd-42ab-4720-9111-a230f6c30c41","Type":"ContainerDied","Data":"a753e877b612535be5308c218a011ee9abb1476648f9226df6a1eb96c5b3d3f5"} Oct 31 13:15:48.037907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.037390 4542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a753e877b612535be5308c218a011ee9abb1476648f9226df6a1eb96c5b3d3f5" Oct 31 13:15:48.038019 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.037969 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:15:48.039303 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.039278 4542 generic.go:356] "Generic (PLEG): container finished" podID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerID="b73524eec74a6cf109978f6d10642882ffa26f38a8be05e8c2113525b3f0c2a9" exitCode=0 Oct 31 13:15:48.039651 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.039631 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" event={"ID":"1f116ed8-404c-4dc1-a81d-364a4cbed2bc","Type":"ContainerDied","Data":"b73524eec74a6cf109978f6d10642882ffa26f38a8be05e8c2113525b3f0c2a9"} Oct 31 13:15:48.039758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.039665 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" event={"ID":"1f116ed8-404c-4dc1-a81d-364a4cbed2bc","Type":"ContainerDied","Data":"77eef2e6fcd560048e3cc58e93b6f7f4b80a8c2cfc9187e2503fb34d6efb8ac6"} Oct 31 13:15:48.039758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.039681 4542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77eef2e6fcd560048e3cc58e93b6f7f4b80a8c2cfc9187e2503fb34d6efb8ac6" Oct 31 13:15:48.046317 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.046295 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:15:48.113346 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.113309 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tls-certs\") pod \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " Oct 31 13:15:48.114287 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.114257 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03559efd-42ab-4720-9111-a230f6c30c41-kube-api-access-zmj2b\" (UniqueName: \"kubernetes.io/projected/03559efd-42ab-4720-9111-a230f6c30c41-kube-api-access-zmj2b\") pod \"03559efd-42ab-4720-9111-a230f6c30c41\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " Oct 31 13:15:48.114821 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.114800 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-tmp-dir\") pod \"03559efd-42ab-4720-9111-a230f6c30c41\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " Oct 31 13:15:48.115111 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.115084 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-tmp-dir" pod "03559efd-42ab-4720-9111-a230f6c30c41" (UID: "03559efd-42ab-4720-9111-a230f6c30c41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:48.115188 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.115169 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-dshm\") pod \"03559efd-42ab-4720-9111-a230f6c30c41\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.115204 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-dshm\") pod \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.115810 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-kube-api-access-cn8b4\" (UniqueName: \"kubernetes.io/projected/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-kube-api-access-cn8b4\") pod \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.115881 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-home\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-home\") pod \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.115940 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tmp-dir\") pod \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.116056 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-model-cache\") pod \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\" (UID: \"1f116ed8-404c-4dc1-a81d-364a4cbed2bc\") " Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.116084 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-home\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-home\") pod \"03559efd-42ab-4720-9111-a230f6c30c41\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.116114 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03559efd-42ab-4720-9111-a230f6c30c41-tls-certs\" (UniqueName: \"kubernetes.io/secret/03559efd-42ab-4720-9111-a230f6c30c41-tls-certs\") pod \"03559efd-42ab-4720-9111-a230f6c30c41\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.116163 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-model-cache\") pod \"03559efd-42ab-4720-9111-a230f6c30c41\" (UID: \"03559efd-42ab-4720-9111-a230f6c30c41\") " Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.116940 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-model-cache" pod "03559efd-42ab-4720-9111-a230f6c30c41" (UID: "03559efd-42ab-4720-9111-a230f6c30c41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.117464 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.117723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.117506 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.118275 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.117815 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-model-cache" pod "1f116ed8-404c-4dc1-a81d-364a4cbed2bc" (UID: "1f116ed8-404c-4dc1-a81d-364a4cbed2bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:48.118275 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.117982 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-home" pod "03559efd-42ab-4720-9111-a230f6c30c41" (UID: "03559efd-42ab-4720-9111-a230f6c30c41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:48.118275 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.118026 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tmp-dir" pod "1f116ed8-404c-4dc1-a81d-364a4cbed2bc" (UID: "1f116ed8-404c-4dc1-a81d-364a4cbed2bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:48.118275 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.118081 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-home" pod "1f116ed8-404c-4dc1-a81d-364a4cbed2bc" (UID: "1f116ed8-404c-4dc1-a81d-364a4cbed2bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:48.122158 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.122129 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-dshm" pod "03559efd-42ab-4720-9111-a230f6c30c41" (UID: "03559efd-42ab-4720-9111-a230f6c30c41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:48.123127 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.123079 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-kube-api-access-cn8b4" pod "1f116ed8-404c-4dc1-a81d-364a4cbed2bc" (UID: "1f116ed8-404c-4dc1-a81d-364a4cbed2bc"). InnerVolumeSpecName "kube-api-access-cn8b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:48.123381 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.123340 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-dshm" pod "1f116ed8-404c-4dc1-a81d-364a4cbed2bc" (UID: "1f116ed8-404c-4dc1-a81d-364a4cbed2bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:48.123381 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.123359 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03559efd-42ab-4720-9111-a230f6c30c41-kube-api-access-zmj2b" pod "03559efd-42ab-4720-9111-a230f6c30c41" (UID: "03559efd-42ab-4720-9111-a230f6c30c41"). InnerVolumeSpecName "kube-api-access-zmj2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:48.124030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.124004 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03559efd-42ab-4720-9111-a230f6c30c41-tls-certs" pod "03559efd-42ab-4720-9111-a230f6c30c41" (UID: "03559efd-42ab-4720-9111-a230f6c30c41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:48.125050 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.125025 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tls-certs" pod "1f116ed8-404c-4dc1-a81d-364a4cbed2bc" (UID: "1f116ed8-404c-4dc1-a81d-364a4cbed2bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:48.219682 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.219634 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.220051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.219898 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.220051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.219925 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.220051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.219942 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.220051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.219958 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03559efd-42ab-4720-9111-a230f6c30c41-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.220051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.219971 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.220051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.219987 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmj2b\" (UniqueName: \"kubernetes.io/projected/03559efd-42ab-4720-9111-a230f6c30c41-kube-api-access-zmj2b\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.220051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.220003 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03559efd-42ab-4720-9111-a230f6c30c41-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.220051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.220016 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.220051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.220029 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn8b4\" (UniqueName: \"kubernetes.io/projected/1f116ed8-404c-4dc1-a81d-364a4cbed2bc-kube-api-access-cn8b4\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.238199 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.238178 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:48.320926 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.320901 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-openshift-ca\") pod \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " Oct 31 13:15:48.321055 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.320999 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-kube-api-access-2zq8w\" (UniqueName: \"kubernetes.io/projected/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-kube-api-access-2zq8w\") pod \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " Oct 31 13:15:48.321106 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.321093 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-xks-ca\") pod \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\" (UID: \"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74\") " Oct 31 13:15:48.321243 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.321221 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-openshift-ca" pod "46bcbb4d-b174-4a1d-b5ce-6e28726bcf74" (UID: "46bcbb4d-b174-4a1d-b5ce-6e28726bcf74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:48.321826 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.321801 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-openshift-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.323190 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.323166 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-xks-ca" pod "46bcbb4d-b174-4a1d-b5ce-6e28726bcf74" (UID: "46bcbb4d-b174-4a1d-b5ce-6e28726bcf74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:48.323275 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.323208 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-kube-api-access-2zq8w" pod "46bcbb4d-b174-4a1d-b5ce-6e28726bcf74" (UID: "46bcbb4d-b174-4a1d-b5ce-6e28726bcf74"). InnerVolumeSpecName "kube-api-access-2zq8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:48.422848 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.422812 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zq8w\" (UniqueName: \"kubernetes.io/projected/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-kube-api-access-2zq8w\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:48.422848 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:48.422846 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74-maas-api-xks-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:49.048192 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.048056 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" Oct 31 13:15:49.048192 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.048084 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l" event={"ID":"46bcbb4d-b174-4a1d-b5ce-6e28726bcf74","Type":"ContainerDied","Data":"19d9ee4941dc71699da795b830b1d569b1e3db2620d756740364efa280a48b31"} Oct 31 13:15:49.048192 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.048144 4542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19d9ee4941dc71699da795b830b1d569b1e3db2620d756740364efa280a48b31" Oct 31 13:15:49.095033 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.048456 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6" Oct 31 13:15:49.095033 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.048933 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p" Oct 31 13:15:49.095033 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.089806 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p"] Oct 31 13:15:49.095033 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.094794 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/simulator-w6-79c6ff-kserve-689c5c5d-h9q9p"] Oct 31 13:15:49.114114 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.114081 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6"] Oct 31 13:15:49.120756 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.120722 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/distinct-2-w6-79c6ff-kserve-fc6c599fc-hz8x6"] Oct 31 13:15:49.295671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.295638 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03559efd-42ab-4720-9111-a230f6c30c41" path="/var/lib/kubelet/pods/03559efd-42ab-4720-9111-a230f6c30c41/volumes" Oct 31 13:15:49.296142 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.296127 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" path="/var/lib/kubelet/pods/1f116ed8-404c-4dc1-a81d-364a4cbed2bc/volumes" Oct 31 13:15:49.525690 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.525658 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk"] Oct 31 13:15:49.530262 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.530232 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm"] Oct 31 13:15:49.534167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:49.534137 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn"] Oct 31 13:15:51.367770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.367735 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc"] Oct 31 13:15:51.368369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368324 4542 cpu_manager.go:414] "containerMap: removing container" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerName="istio-proxy" Oct 31 13:15:51.368369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368336 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerName="istio-proxy" Oct 31 13:15:51.368369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368348 4542 cpu_manager.go:414] "containerMap: removing container" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerName="cleanup" Oct 31 13:15:51.368369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368353 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerName="cleanup" Oct 31 13:15:51.368369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368368 4542 cpu_manager.go:414] "containerMap: removing container" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" Oct 31 13:15:51.368369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368373 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368385 4542 cpu_manager.go:414] "containerMap: removing container" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerName="cleanup" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368389 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerName="cleanup" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368403 4542 cpu_manager.go:414] "containerMap: removing container" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368407 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368416 4542 cpu_manager.go:414] "containerMap: removing container" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368420 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368430 4542 cpu_manager.go:414] "containerMap: removing container" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368434 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368445 4542 cpu_manager.go:414] "containerMap: removing container" podUID="46bcbb4d-b174-4a1d-b5ce-6e28726bcf74" containerName="revoke-keys" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368449 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bcbb4d-b174-4a1d-b5ce-6e28726bcf74" containerName="revoke-keys" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368467 4542 cpu_manager.go:414] "containerMap: removing container" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368484 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368562 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="03559efd-42ab-4720-9111-a230f6c30c41" containerName="main" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368573 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de77bc5-25f8-4e11-ba04-f3f254a594fc" containerName="istio-proxy" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368580 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a79ba0-5690-4754-8ad0-8689bcaa1ae8" containerName="cleanup" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368586 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368593 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecdecd31-e480-4f1d-86c4-a023cc2e418e" containerName="cleanup" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368600 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f116ed8-404c-4dc1-a81d-364a4cbed2bc" containerName="main" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368608 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0d0c59-d655-4733-993c-d89bca07faf9" containerName="istio-proxy" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368615 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cb27e2-2458-4c4f-8b00-b145c9bd8116" containerName="istio-proxy" Oct 31 13:15:51.368708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.368623 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bcbb4d-b174-4a1d-b5ce-6e28726bcf74" containerName="revoke-keys" Oct 31 13:15:51.375653 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.375631 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.385867 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.385839 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\"" Oct 31 13:15:51.394580 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.390464 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:15:51.397529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.397504 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc"] Oct 31 13:15:51.435368 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.435334 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ec18d26-58klc\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.435543 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.435409 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw5qp\" (UniqueName: \"kubernetes.io/projected/6d9922f9-afbe-4381-85a8-7c77c251cfba-kube-api-access-jw5qp\") pod \"maas-api-revoke-sub-0ec18d26-58klc\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.435621 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.435556 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ec18d26-58klc\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.576544 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.559857 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ec18d26-58klc\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.593082 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.593050 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw5qp\" (UniqueName: \"kubernetes.io/projected/6d9922f9-afbe-4381-85a8-7c77c251cfba-kube-api-access-jw5qp\") pod \"maas-api-revoke-sub-0ec18d26-58klc\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.593414 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.593392 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ec18d26-58klc\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.594124 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.592813 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ec18d26-58klc\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.638386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.624227 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw5qp\" (UniqueName: \"kubernetes.io/projected/6d9922f9-afbe-4381-85a8-7c77c251cfba-kube-api-access-jw5qp\") pod \"maas-api-revoke-sub-0ec18d26-58klc\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.641962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.641929 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ec18d26-58klc\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.699813 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.699769 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:51.992402 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:51.992370 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc"] Oct 31 13:15:51.992916 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:15:51.992887 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9922f9_afbe_4381_85a8_7c77c251cfba.slice/crio-f94b73c6ba1f4ce8c381fcee57d152c62b0a0c46c787aaece067ddef119c5c1b WatchSource:0}: Error finding container f94b73c6ba1f4ce8c381fcee57d152c62b0a0c46c787aaece067ddef119c5c1b: Status 404 returned error can't find the container with id f94b73c6ba1f4ce8c381fcee57d152c62b0a0c46c787aaece067ddef119c5c1b Oct 31 13:15:52.067459 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.067429 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" event={"ID":"6d9922f9-afbe-4381-85a8-7c77c251cfba","Type":"ContainerStarted","Data":"f94b73c6ba1f4ce8c381fcee57d152c62b0a0c46c787aaece067ddef119c5c1b"} Oct 31 13:15:52.157524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.157494 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs"] Oct 31 13:15:52.162923 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.162907 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.180120 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.180089 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs"] Oct 31 13:15:52.305747 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.305717 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.305927 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.305754 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.306003 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.305934 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c55sn\" (UniqueName: \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-kube-api-access-c55sn\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.306003 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.305982 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.306083 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.306059 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.306118 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.306091 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.306118 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.306109 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.306228 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.306152 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.306228 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.306216 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.407647 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.407615 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408088 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.407814 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408088 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.407858 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408088 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.408050 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408326 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.408165 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c55sn\" (UniqueName: \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-kube-api-access-c55sn\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408326 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.408173 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408326 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.408242 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408564 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.408339 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408564 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.408376 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408564 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.408402 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408564 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.408440 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.408616 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.408862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.408849 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.409143 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.409110 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.410352 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.410331 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.410809 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.410788 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.424078 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.424057 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.424399 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.424382 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c55sn\" (UniqueName: \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-kube-api-access-c55sn\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.482744 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.482722 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:52.776127 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.776094 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs"] Oct 31 13:15:52.776434 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:15:52.776408 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48e0675_ee4f_4089_adcd_8c6eff556d2d.slice/crio-6e4bc3f9469ab3ebae68c2283ea9a5fd3035339e009912fb701ab0067e52f27e WatchSource:0}: Error finding container 6e4bc3f9469ab3ebae68c2283ea9a5fd3035339e009912fb701ab0067e52f27e: Status 404 returned error can't find the container with id 6e4bc3f9469ab3ebae68c2283ea9a5fd3035339e009912fb701ab0067e52f27e Oct 31 13:15:52.779097 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.779063 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:15:52.779208 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.779143 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:15:52.779208 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:52.779185 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:15:53.077325 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:53.077286 4542 generic.go:356] "Generic (PLEG): container finished" podID="6d9922f9-afbe-4381-85a8-7c77c251cfba" containerID="88efee26a9e941d97ed20f998e5f163ce2d5e19306643266893a15ecb3b9b89a" exitCode=0 Oct 31 13:15:53.077529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:53.077363 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" event={"ID":"6d9922f9-afbe-4381-85a8-7c77c251cfba","Type":"ContainerDied","Data":"88efee26a9e941d97ed20f998e5f163ce2d5e19306643266893a15ecb3b9b89a"} Oct 31 13:15:53.079322 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:53.079284 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" event={"ID":"c48e0675-ee4f-4089-adcd-8c6eff556d2d","Type":"ContainerStarted","Data":"d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21"} Oct 31 13:15:53.079433 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:53.079348 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" event={"ID":"c48e0675-ee4f-4089-adcd-8c6eff556d2d","Type":"ContainerStarted","Data":"6e4bc3f9469ab3ebae68c2283ea9a5fd3035339e009912fb701ab0067e52f27e"} Oct 31 13:15:53.130002 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:53.129955 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podStartSLOduration=1.129940928 podStartE2EDuration="1.129940928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:53.127230988 +0000 UTC m=+969.972397419" watchObservedRunningTime="2026-10-31 13:15:53.129940928 +0000 UTC m=+969.975107436" Oct 31 13:15:53.226803 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:53.226766 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:15:53.483231 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:53.483150 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:15:53.484889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:53.484867 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:15:53.485021 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:53.484922 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 13:15:53.833564 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:15:53.833530 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:15:54.348500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:54.347626 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" podUID="c3f742af-293a-40d6-9a93-75e3030306b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:15:54.623847 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:54.623810 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:15:54.880706 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:54.845656 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 13:15:55.074232 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.074191 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs"] Oct 31 13:15:55.074939 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.074908 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" containerID="cri-o://cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70" gracePeriod=30 Oct 31 13:15:55.556559 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.528592 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:15:55.556559 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.528687 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 13:15:55.655502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.655254 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq"] Oct 31 13:15:55.679501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.671713 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" containerID="cri-o://299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3" gracePeriod=30 Oct 31 13:15:55.740063 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.740014 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:55.804513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.784693 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:55.899545 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.883440 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:55.899545 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.884986 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:55.900633 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.900596 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:55.953562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:55.934700 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:15:56.078664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.070126 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9922f9-afbe-4381-85a8-7c77c251cfba-kube-api-access-jw5qp\" (UniqueName: \"kubernetes.io/projected/6d9922f9-afbe-4381-85a8-7c77c251cfba-kube-api-access-jw5qp\") pod \"6d9922f9-afbe-4381-85a8-7c77c251cfba\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " Oct 31 13:15:56.078664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.070359 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-openshift-ca\") pod \"6d9922f9-afbe-4381-85a8-7c77c251cfba\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " Oct 31 13:15:56.078664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.070419 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-xks-ca\") pod \"6d9922f9-afbe-4381-85a8-7c77c251cfba\" (UID: \"6d9922f9-afbe-4381-85a8-7c77c251cfba\") " Oct 31 13:15:56.110733 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.110697 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-openshift-ca" pod "6d9922f9-afbe-4381-85a8-7c77c251cfba" (UID: "6d9922f9-afbe-4381-85a8-7c77c251cfba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:56.205995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.205887 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-openshift-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:56.280731 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.280687 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-xks-ca" pod "6d9922f9-afbe-4381-85a8-7c77c251cfba" (UID: "6d9922f9-afbe-4381-85a8-7c77c251cfba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:56.385501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.344614 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9922f9-afbe-4381-85a8-7c77c251cfba-maas-api-xks-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:56.385760 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.356758 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9922f9-afbe-4381-85a8-7c77c251cfba-kube-api-access-jw5qp" pod "6d9922f9-afbe-4381-85a8-7c77c251cfba" (UID: "6d9922f9-afbe-4381-85a8-7c77c251cfba"). InnerVolumeSpecName "kube-api-access-jw5qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:56.577562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.529101 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw5qp\" (UniqueName: \"kubernetes.io/projected/6d9922f9-afbe-4381-85a8-7c77c251cfba-kube-api-access-jw5qp\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:56.607562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.543602 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:15:56.607865 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.607832 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 13:15:56.810507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.809622 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:56.810507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.810065 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" Oct 31 13:15:56.811074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.810686 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:15:56.811074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.810723 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc" event={"ID":"6d9922f9-afbe-4381-85a8-7c77c251cfba","Type":"ContainerDied","Data":"f94b73c6ba1f4ce8c381fcee57d152c62b0a0c46c787aaece067ddef119c5c1b"} Oct 31 13:15:56.811074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.810758 4542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f94b73c6ba1f4ce8c381fcee57d152c62b0a0c46c787aaece067ddef119c5c1b" Oct 31 13:15:56.898502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.897852 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:56.898502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.898251 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:56.918499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.898900 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:15:56.918499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.898964 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:15:56.918499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.900942 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:56.918499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:56.901263 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:15:57.552500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:57.548806 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:15:57.552500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:57.548894 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 13:15:57.552500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:57.549337 4542 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:57.820552 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:57.812557 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 31 13:15:57.862537 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:57.837256 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:57.927517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:57.898726 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:57.927517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:57.901806 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 13:15:57.939589 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:57.935962 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:58.549501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:58.548513 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:15:58.549501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:58.548578 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 13:15:58.823750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:58.823706 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" podUID="1d5dc13e-a93d-4867-bd22-ccf73246c370" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:58.851829 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:58.843370 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:15:58.862941 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:58.862615 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:15:58.909007 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:58.908965 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:58.926758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:58.926707 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 13:15:58.942274 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:58.942242 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 13:15:59.033655 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.033581 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:15:59.034783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.034755 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj" Oct 31 13:15:59.034901 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.034871 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:15:59.035814 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.035794 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:15:59.044671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.044651 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:15:59.201835 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.201746 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds"] Oct 31 13:15:59.202602 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.202568 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" podUID="2541cf2f-8571-49cc-995f-4174130110be" containerName="maas-api" containerID="cri-o://ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e" gracePeriod=30 Oct 31 13:15:59.491645 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.491108 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:15:59.491645 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.491170 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 13:15:59.632822 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.632789 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:15:59.773576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.772861 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2541cf2f-8571-49cc-995f-4174130110be-kube-api-access-zfdl2\" (UniqueName: \"kubernetes.io/projected/2541cf2f-8571-49cc-995f-4174130110be-kube-api-access-zfdl2\") pod \"2541cf2f-8571-49cc-995f-4174130110be\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " Oct 31 13:15:59.773576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.772932 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2541cf2f-8571-49cc-995f-4174130110be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2541cf2f-8571-49cc-995f-4174130110be-openshift-service-ca\") pod \"2541cf2f-8571-49cc-995f-4174130110be\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " Oct 31 13:15:59.773576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.773005 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-metrics-certs\") pod \"2541cf2f-8571-49cc-995f-4174130110be\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " Oct 31 13:15:59.773576 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.773028 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-maas-api-tls\") pod \"2541cf2f-8571-49cc-995f-4174130110be\" (UID: \"2541cf2f-8571-49cc-995f-4174130110be\") " Oct 31 13:15:59.774306 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.774274 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2541cf2f-8571-49cc-995f-4174130110be-openshift-service-ca" pod "2541cf2f-8571-49cc-995f-4174130110be" (UID: "2541cf2f-8571-49cc-995f-4174130110be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:59.778247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.777728 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2541cf2f-8571-49cc-995f-4174130110be-kube-api-access-zfdl2" pod "2541cf2f-8571-49cc-995f-4174130110be" (UID: "2541cf2f-8571-49cc-995f-4174130110be"). InnerVolumeSpecName "kube-api-access-zfdl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:59.778392 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.778286 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-maas-api-tls" pod "2541cf2f-8571-49cc-995f-4174130110be" (UID: "2541cf2f-8571-49cc-995f-4174130110be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:59.778705 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.778678 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-metrics-certs" pod "2541cf2f-8571-49cc-995f-4174130110be" (UID: "2541cf2f-8571-49cc-995f-4174130110be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:15:59.880499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.878136 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:15:59.880499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.878207 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:15:59.880499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.878784 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfdl2\" (UniqueName: \"kubernetes.io/projected/2541cf2f-8571-49cc-995f-4174130110be-kube-api-access-zfdl2\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:59.880499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.878823 4542 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2541cf2f-8571-49cc-995f-4174130110be-openshift-service-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:59.880499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.878839 4542 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-metrics-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:59.880499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:15:59.878854 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2541cf2f-8571-49cc-995f-4174130110be-maas-api-tls\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:00.015063 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.015026 4542 generic.go:356] "Generic (PLEG): container finished" podID="2541cf2f-8571-49cc-995f-4174130110be" containerID="ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e" exitCode=0 Oct 31 13:16:00.015259 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.015079 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" Oct 31 13:16:00.015259 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.015110 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" event={"ID":"2541cf2f-8571-49cc-995f-4174130110be","Type":"ContainerDied","Data":"ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e"} Oct 31 13:16:00.015259 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.015176 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds" event={"ID":"2541cf2f-8571-49cc-995f-4174130110be","Type":"ContainerDied","Data":"04be76536f20f8e2fafb25b9e80a99cb679d2d5160eec69c31c1b47017daf781"} Oct 31 13:16:00.015259 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.015207 4542 scope.go:122] "RemoveContainer" containerID="ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e" Oct 31 13:16:00.026419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.026396 4542 scope.go:122] "RemoveContainer" containerID="ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e" Oct 31 13:16:00.026741 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:00.026721 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e\": container with ID starting with ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e not found: ID does not exist" containerID="ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e" Oct 31 13:16:00.026825 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.026754 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e"} err="failed to get container status \"ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e\": rpc error: code = NotFound desc = could not find container \"ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e\": container with ID starting with ad00464c38f289379845988c6ab66528dafeae3c222c881e5c3e786d317ab99e not found: ID does not exist" Oct 31 13:16:00.069488 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.069436 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds"] Oct 31 13:16:00.072899 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.072872 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-79c6ff-5d8dfd6dcd-hstds"] Oct 31 13:16:00.369732 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.369688 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" containerID="cri-o://0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0" gracePeriod=45 Oct 31 13:16:00.483237 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.483210 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:16:00.483370 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.483259 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 13:16:00.671112 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.671024 4542 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:00.671112 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.671099 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:00.710113 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.710089 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:16:00.804575 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.804547 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:16:00.897096 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.897060 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tls-certs\") pod \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " Oct 31 13:16:00.897511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.897116 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tmp-dir\") pod \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " Oct 31 13:16:00.897511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.897253 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-home\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-home\") pod \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " Oct 31 13:16:00.897511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.897294 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-dshm\") pod \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " Oct 31 13:16:00.897511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.897382 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tmp-dir" pod "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" (UID: "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:00.897511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.897470 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-kube-api-access-cw5fn\" (UniqueName: \"kubernetes.io/projected/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-kube-api-access-cw5fn\") pod \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " Oct 31 13:16:00.897790 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.897513 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-home" pod "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" (UID: "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:00.897790 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.897528 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-model-cache\") pod \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\" (UID: \"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1\") " Oct 31 13:16:00.897790 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.897771 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-model-cache" pod "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" (UID: "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:00.898035 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.898016 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:00.898098 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.898035 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:00.898098 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.898049 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:00.899572 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.899540 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-dshm" pod "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" (UID: "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:00.899572 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.899558 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tls-certs" pod "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" (UID: "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:16:00.899759 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.899570 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-kube-api-access-cw5fn" pod "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" (UID: "1f3ae33d-ed8c-4ce2-ba24-53be34159ba1"). InnerVolumeSpecName "kube-api-access-cw5fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:00.999367 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.999283 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-socket\") pod \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " Oct 31 13:16:00.999367 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.999327 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-envoy\") pod \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " Oct 31 13:16:00.999606 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.999375 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-certs\") pod \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " Oct 31 13:16:00.999606 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.999457 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-kube-api-access-z7xgq\" (UniqueName: \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-kube-api-access-z7xgq\") pod \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " Oct 31 13:16:00.999606 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.999553 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-data\") pod \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " Oct 31 13:16:00.999758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.999615 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istiod-ca-cert\") pod \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " Oct 31 13:16:00.999758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.999685 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-podinfo\") pod \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " Oct 31 13:16:00.999859 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.999801 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-token\" (UniqueName: \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-token\") pod \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " Oct 31 13:16:00.999859 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.999844 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-credential-socket\") pod \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\" (UID: \"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee\") " Oct 31 13:16:01.000019 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:00.999983 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-data" pod "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" (UID: "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:01.000092 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.000064 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istiod-ca-cert" pod "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" (UID: "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:01.000347 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.000203 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-certs" pod "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" (UID: "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:01.000347 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.000313 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-credential-socket" pod "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" (UID: "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:01.000754 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.000731 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-socket" pod "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" (UID: "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:01.000904 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.000742 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.001034 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.001020 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.001154 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.001140 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw5fn\" (UniqueName: \"kubernetes.io/projected/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-kube-api-access-cw5fn\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.001282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.001268 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.001401 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.001388 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.001548 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.001533 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.001674 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.001661 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.002057 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.002039 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-token" pod "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" (UID: "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:01.002617 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.002588 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-podinfo" pod "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" (UID: "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:01.003557 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.003534 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-envoy" pod "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" (UID: "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:01.003723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.003700 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-kube-api-access-z7xgq" pod "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" (UID: "7c7c3b8b-6f23-4fe9-be0e-42b98613bcee"). InnerVolumeSpecName "kube-api-access-z7xgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:01.024294 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.024258 4542 generic.go:356] "Generic (PLEG): container finished" podID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerID="0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0" exitCode=0 Oct 31 13:16:01.024434 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.024297 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" Oct 31 13:16:01.024434 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.024339 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" event={"ID":"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1","Type":"ContainerDied","Data":"0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0"} Oct 31 13:16:01.024434 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.024383 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg" event={"ID":"1f3ae33d-ed8c-4ce2-ba24-53be34159ba1","Type":"ContainerDied","Data":"fbfd139c64a3f12cd07a56c622386f78c5694039ee93cf1087f76b7f3ba62d9f"} Oct 31 13:16:01.024434 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.024409 4542 scope.go:122] "RemoveContainer" containerID="0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0" Oct 31 13:16:01.027521 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.027465 4542 generic.go:356] "Generic (PLEG): container finished" podID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerID="cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70" exitCode=0 Oct 31 13:16:01.027624 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.027518 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" event={"ID":"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee","Type":"ContainerDied","Data":"cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70"} Oct 31 13:16:01.027624 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.027562 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" Oct 31 13:16:01.027624 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.027576 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs" event={"ID":"7c7c3b8b-6f23-4fe9-be0e-42b98613bcee","Type":"ContainerDied","Data":"4d5b2bd1c72cc0ced75e0c796c86967e7208fa961356ee2b8d74fc4878ca946d"} Oct 31 13:16:01.035207 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.035187 4542 scope.go:122] "RemoveContainer" containerID="0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0" Oct 31 13:16:01.035618 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:01.035537 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0\": container with ID starting with 0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0 not found: ID does not exist" containerID="0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0" Oct 31 13:16:01.035618 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.035578 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0"} err="failed to get container status \"0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0\": rpc error: code = NotFound desc = could not find container \"0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0\": container with ID starting with 0cc98d324afc2f43f42dde842b1c7dc8406675a1b5b3dd255176cad731ba54d0 not found: ID does not exist" Oct 31 13:16:01.035618 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.035604 4542 scope.go:122] "RemoveContainer" containerID="cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70" Oct 31 13:16:01.051600 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.051576 4542 scope.go:122] "RemoveContainer" containerID="fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6" Oct 31 13:16:01.061688 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.061663 4542 scope.go:122] "RemoveContainer" containerID="cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70" Oct 31 13:16:01.062017 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:01.061997 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70\": container with ID starting with cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70 not found: ID does not exist" containerID="cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70" Oct 31 13:16:01.062103 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.062050 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70"} err="failed to get container status \"cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70\": rpc error: code = NotFound desc = could not find container \"cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70\": container with ID starting with cbdfb0a59c13c8b565b34c6adf0d734876d9b5cfedb98032c9646e99a2fa0d70 not found: ID does not exist" Oct 31 13:16:01.062103 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.062070 4542 scope.go:122] "RemoveContainer" containerID="fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6" Oct 31 13:16:01.062393 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:01.062376 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6\": container with ID starting with fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6 not found: ID does not exist" containerID="fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6" Oct 31 13:16:01.062453 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.062399 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6"} err="failed to get container status \"fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6\": rpc error: code = NotFound desc = could not find container \"fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6\": container with ID starting with fcdf8949ef7885ae0ba1b7039fdcad3e6a00d870f42991efddcc9d4e371ba7f6 not found: ID does not exist" Oct 31 13:16:01.064321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.064300 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg"] Oct 31 13:16:01.067366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.067342 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-w4-426787-kserve-5d5c54f69-7xpkg"] Oct 31 13:16:01.085176 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.085142 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs"] Oct 31 13:16:01.092220 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.092196 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-lcpbs"] Oct 31 13:16:01.103106 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.103074 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.103205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.103111 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.103205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.103130 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7xgq\" (UniqueName: \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-kube-api-access-z7xgq\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.103205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.103147 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.103205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.103161 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.310643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.308606 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" path="/var/lib/kubelet/pods/1f3ae33d-ed8c-4ce2-ba24-53be34159ba1/volumes" Oct 31 13:16:01.310643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.309244 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2541cf2f-8571-49cc-995f-4174130110be" path="/var/lib/kubelet/pods/2541cf2f-8571-49cc-995f-4174130110be/volumes" Oct 31 13:16:01.310643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.309901 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" path="/var/lib/kubelet/pods/7c7c3b8b-6f23-4fe9-be0e-42b98613bcee/volumes" Oct 31 13:16:01.398668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.398301 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.406958 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-socket\") pod \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407003 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-data\") pod \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407055 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-envoy\") pod \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407076 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istiod-ca-cert\") pod \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407115 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-credential-socket\") pod \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407131 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-token\" (UniqueName: \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-token\") pod \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407167 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-podinfo\") pod \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407258 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-certs\") pod \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407415 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-kube-api-access-fvkxc\" (UniqueName: \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-kube-api-access-fvkxc\") pod \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\" (UID: \"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0\") " Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407415 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-credential-socket" pod "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" (UID: "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407430 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-data" pod "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" (UID: "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:01.407935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.407665 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-certs" pod "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" (UID: "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:01.408742 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.408070 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.408742 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.408095 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.408742 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.408111 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.408742 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.408137 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istiod-ca-cert" pod "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" (UID: "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:01.408742 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.408581 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-socket" pod "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" (UID: "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:01.410085 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.410031 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-token" pod "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" (UID: "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:01.410339 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.410281 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-podinfo" pod "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" (UID: "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:01.412405 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.412376 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-kube-api-access-fvkxc" pod "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" (UID: "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0"). InnerVolumeSpecName "kube-api-access-fvkxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:01.413563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.413183 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-envoy" pod "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" (UID: "e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:01.483368 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.483329 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:16:01.483605 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.483399 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 13:16:01.509946 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.509912 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.509946 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.509941 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.510177 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.509953 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.510177 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.509965 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.510177 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.509974 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvkxc\" (UniqueName: \"kubernetes.io/projected/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-kube-api-access-fvkxc\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:01.510177 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:01.509983 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:02.038886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.038846 4542 generic.go:356] "Generic (PLEG): container finished" podID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerID="299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3" exitCode=0 Oct 31 13:16:02.039389 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.038897 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" Oct 31 13:16:02.039389 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.038930 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" event={"ID":"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0","Type":"ContainerDied","Data":"299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3"} Oct 31 13:16:02.039389 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.039001 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq" event={"ID":"e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0","Type":"ContainerDied","Data":"5f583486424f88365bd71b4861246a677d9460e3be51d270ca3f0570677ea215"} Oct 31 13:16:02.039389 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.039034 4542 scope.go:122] "RemoveContainer" containerID="299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3" Oct 31 13:16:02.052464 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.052442 4542 scope.go:122] "RemoveContainer" containerID="299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3" Oct 31 13:16:02.052834 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:02.052805 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3\": container with ID starting with 299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3 not found: ID does not exist" containerID="299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3" Oct 31 13:16:02.052929 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.052845 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3"} err="failed to get container status \"299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3\": rpc error: code = NotFound desc = could not find container \"299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3\": container with ID starting with 299c4e846a550e96a8fb7abd6f2a0bed2891d0181a0a27bb1114b702538483b3 not found: ID does not exist" Oct 31 13:16:02.095055 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.095017 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq"] Oct 31 13:16:02.100855 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.100820 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-vkjkq"] Oct 31 13:16:02.860015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.859978 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk"] Oct 31 13:16:02.860199 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:02.860180 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" podUID="ff621f22-5195-4616-9fb0-3604409740e7" containerName="maas-api" containerID="cri-o://2df4812e0f22adf1d00adb8d4f96a016ef6ac8c7e2c36464cdbdc8108c879d1a" gracePeriod=30 Oct 31 13:16:03.108835 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.108785 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" podUID="ff621f22-5195-4616-9fb0-3604409740e7" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.40:8443/health\": dial tcp 10.133.0.40:8443: connect: connection refused" Oct 31 13:16:03.305289 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.305253 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" path="/var/lib/kubelet/pods/e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0/volumes" Oct 31 13:16:03.566499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.565701 4542 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:16:03.566499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.565787 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:16:03.846109 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.846077 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:16:03.896653 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.896611 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3f742af-293a-40d6-9a93-75e3030306b8-kube-api-access-8ptbj\" (UniqueName: \"kubernetes.io/projected/c3f742af-293a-40d6-9a93-75e3030306b8-kube-api-access-8ptbj\") pod \"c3f742af-293a-40d6-9a93-75e3030306b8\" (UID: \"c3f742af-293a-40d6-9a93-75e3030306b8\") " Oct 31 13:16:03.896827 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.896740 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3f742af-293a-40d6-9a93-75e3030306b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f742af-293a-40d6-9a93-75e3030306b8-plugins-config-volume\") pod \"c3f742af-293a-40d6-9a93-75e3030306b8\" (UID: \"c3f742af-293a-40d6-9a93-75e3030306b8\") " Oct 31 13:16:03.897615 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.897569 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f742af-293a-40d6-9a93-75e3030306b8-plugins-config-volume" pod "c3f742af-293a-40d6-9a93-75e3030306b8" (UID: "c3f742af-293a-40d6-9a93-75e3030306b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:03.899412 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.899386 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f742af-293a-40d6-9a93-75e3030306b8-kube-api-access-8ptbj" pod "c3f742af-293a-40d6-9a93-75e3030306b8" (UID: "c3f742af-293a-40d6-9a93-75e3030306b8"). InnerVolumeSpecName "kube-api-access-8ptbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:03.998987 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.998946 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ptbj\" (UniqueName: \"kubernetes.io/projected/c3f742af-293a-40d6-9a93-75e3030306b8-kube-api-access-8ptbj\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:03.999188 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:03.999170 4542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f742af-293a-40d6-9a93-75e3030306b8-plugins-config-volume\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.013593 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.013370 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:16:04.014214 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.014188 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:16:04.015373 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.015342 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:16:04.057992 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.057963 4542 generic.go:356] "Generic (PLEG): container finished" podID="ff621f22-5195-4616-9fb0-3604409740e7" containerID="2df4812e0f22adf1d00adb8d4f96a016ef6ac8c7e2c36464cdbdc8108c879d1a" exitCode=0 Oct 31 13:16:04.058139 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.058000 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" event={"ID":"ff621f22-5195-4616-9fb0-3604409740e7","Type":"ContainerDied","Data":"2df4812e0f22adf1d00adb8d4f96a016ef6ac8c7e2c36464cdbdc8108c879d1a"} Oct 31 13:16:04.059472 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.059446 4542 generic.go:356] "Generic (PLEG): container finished" podID="c3f742af-293a-40d6-9a93-75e3030306b8" containerID="67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6" exitCode=0 Oct 31 13:16:04.059600 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.059520 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" event={"ID":"c3f742af-293a-40d6-9a93-75e3030306b8","Type":"ContainerDied","Data":"67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6"} Oct 31 13:16:04.059600 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.059540 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" Oct 31 13:16:04.059600 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.059573 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx" event={"ID":"c3f742af-293a-40d6-9a93-75e3030306b8","Type":"ContainerDied","Data":"cbaa1fb8073973c3de8764265c88cd475d956f1dcc01e9f2019b4bb1266347cb"} Oct 31 13:16:04.059763 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.059605 4542 scope.go:122] "RemoveContainer" containerID="67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6" Oct 31 13:16:04.073072 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.072765 4542 scope.go:122] "RemoveContainer" containerID="67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6" Oct 31 13:16:04.075004 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:04.074938 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6\": container with ID starting with 67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6 not found: ID does not exist" containerID="67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6" Oct 31 13:16:04.075004 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.074971 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6"} err="failed to get container status \"67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6\": rpc error: code = NotFound desc = could not find container \"67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6\": container with ID starting with 67feab458489749a6262fff2a4a2b162a93ea300716c63f3b704112e337fbfc6 not found: ID does not exist" Oct 31 13:16:04.099632 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.099551 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx"] Oct 31 13:16:04.104347 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.104317 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-79c6ff-557dff8499-smxmx"] Oct 31 13:16:04.132122 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.132070 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:16:04.202194 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.202165 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff621f22-5195-4616-9fb0-3604409740e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff621f22-5195-4616-9fb0-3604409740e7-openshift-service-ca\") pod \"ff621f22-5195-4616-9fb0-3604409740e7\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " Oct 31 13:16:04.202338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.202200 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff621f22-5195-4616-9fb0-3604409740e7-kube-api-access-xbr9x\" (UniqueName: \"kubernetes.io/projected/ff621f22-5195-4616-9fb0-3604409740e7-kube-api-access-xbr9x\") pod \"ff621f22-5195-4616-9fb0-3604409740e7\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " Oct 31 13:16:04.202338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.202246 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-metrics-certs\") pod \"ff621f22-5195-4616-9fb0-3604409740e7\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " Oct 31 13:16:04.202338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.202325 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-maas-api-tls\") pod \"ff621f22-5195-4616-9fb0-3604409740e7\" (UID: \"ff621f22-5195-4616-9fb0-3604409740e7\") " Oct 31 13:16:04.202650 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.202625 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff621f22-5195-4616-9fb0-3604409740e7-openshift-service-ca" pod "ff621f22-5195-4616-9fb0-3604409740e7" (UID: "ff621f22-5195-4616-9fb0-3604409740e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:04.203175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.203106 4542 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff621f22-5195-4616-9fb0-3604409740e7-openshift-service-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.204563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.204541 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-metrics-certs" pod "ff621f22-5195-4616-9fb0-3604409740e7" (UID: "ff621f22-5195-4616-9fb0-3604409740e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:16:04.204635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.204569 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-maas-api-tls" pod "ff621f22-5195-4616-9fb0-3604409740e7" (UID: "ff621f22-5195-4616-9fb0-3604409740e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:16:04.204809 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.204789 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff621f22-5195-4616-9fb0-3604409740e7-kube-api-access-xbr9x" pod "ff621f22-5195-4616-9fb0-3604409740e7" (UID: "ff621f22-5195-4616-9fb0-3604409740e7"). InnerVolumeSpecName "kube-api-access-xbr9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:04.304407 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.304366 4542 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-metrics-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.304407 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.304405 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff621f22-5195-4616-9fb0-3604409740e7-maas-api-tls\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.304603 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.304418 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbr9x\" (UniqueName: \"kubernetes.io/projected/ff621f22-5195-4616-9fb0-3604409740e7-kube-api-access-xbr9x\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.551429 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.551328 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" containerID="cri-o://2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40" gracePeriod=45 Oct 31 13:16:04.554825 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.554789 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" containerID="cri-o://5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a" gracePeriod=45 Oct 31 13:16:04.561992 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.561959 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" containerID="cri-o://42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2" gracePeriod=45 Oct 31 13:16:04.835529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.835466 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:16:04.853507 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.853398 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:16:04.906412 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.906376 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:16:04.922836 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.922801 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tls-certs\") pod \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " Oct 31 13:16:04.923119 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.923094 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tmp-dir\") pod \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " Oct 31 13:16:04.923119 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.923140 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tmp-dir\") pod \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " Oct 31 13:16:04.923401 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.923192 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-model-cache\") pod \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " Oct 31 13:16:04.923559 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.923516 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tmp-dir" pod "5c11b2df-1fc1-40ca-86f0-315f19e683e7" (UID: "5c11b2df-1fc1-40ca-86f0-315f19e683e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:04.923711 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.923672 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-model-cache" pod "5c11b2df-1fc1-40ca-86f0-315f19e683e7" (UID: "5c11b2df-1fc1-40ca-86f0-315f19e683e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:04.923765 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.923730 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tmp-dir" pod "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" (UID: "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:04.923825 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.923808 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-home\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-home\") pod \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " Oct 31 13:16:04.924058 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.924037 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-home" pod "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" (UID: "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:04.924139 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.924119 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c11b2df-1fc1-40ca-86f0-315f19e683e7-kube-api-access-4nq29\" (UniqueName: \"kubernetes.io/projected/5c11b2df-1fc1-40ca-86f0-315f19e683e7-kube-api-access-4nq29\") pod \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " Oct 31 13:16:04.924207 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.924190 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-model-cache\") pod \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " Oct 31 13:16:04.924262 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.924233 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-dshm\") pod \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " Oct 31 13:16:04.924313 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.924277 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tls-certs\" (UniqueName: \"kubernetes.io/secret/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tls-certs\") pod \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " Oct 31 13:16:04.924313 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.924304 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-home\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-home\") pod \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\" (UID: \"5c11b2df-1fc1-40ca-86f0-315f19e683e7\") " Oct 31 13:16:04.924419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.924341 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-kube-api-access-kf2h4\" (UniqueName: \"kubernetes.io/projected/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-kube-api-access-kf2h4\") pod \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " Oct 31 13:16:04.924419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.924391 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-dshm\") pod \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\" (UID: \"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937\") " Oct 31 13:16:04.924892 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.924870 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-model-cache" pod "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" (UID: "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:04.926087 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.925828 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-home" pod "5c11b2df-1fc1-40ca-86f0-315f19e683e7" (UID: "5c11b2df-1fc1-40ca-86f0-315f19e683e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:04.927135 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.927115 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.927246 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.927227 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.927340 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.927249 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.927340 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.927263 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.927340 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.927275 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.927340 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.927287 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:04.927957 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.927934 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-dshm" pod "5c11b2df-1fc1-40ca-86f0-315f19e683e7" (UID: "5c11b2df-1fc1-40ca-86f0-315f19e683e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:04.928154 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.928127 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tls-certs" pod "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" (UID: "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:16:04.930267 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.930230 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-kube-api-access-kf2h4" pod "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" (UID: "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937"). InnerVolumeSpecName "kube-api-access-kf2h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:04.930359 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.930298 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c11b2df-1fc1-40ca-86f0-315f19e683e7-kube-api-access-4nq29" pod "5c11b2df-1fc1-40ca-86f0-315f19e683e7" (UID: "5c11b2df-1fc1-40ca-86f0-315f19e683e7"). InnerVolumeSpecName "kube-api-access-4nq29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:04.932170 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.932144 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-dshm" pod "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" (UID: "be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:04.939521 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:04.932463 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tls-certs" pod "5c11b2df-1fc1-40ca-86f0-315f19e683e7" (UID: "5c11b2df-1fc1-40ca-86f0-315f19e683e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:16:05.028084 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.028054 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-home\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-home\") pod \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " Oct 31 13:16:05.028231 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.028153 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-kube-api-access-lhrr6\" (UniqueName: \"kubernetes.io/projected/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-kube-api-access-lhrr6\") pod \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " Oct 31 13:16:05.028231 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.028185 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tls-certs\") pod \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " Oct 31 13:16:05.028354 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.028250 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-dshm\") pod \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " Oct 31 13:16:05.028354 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.028297 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-home" pod "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" (UID: "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:05.028527 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.028420 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tmp-dir\") pod \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " Oct 31 13:16:05.028527 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.028462 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-model-cache\") pod \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\" (UID: \"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2\") " Oct 31 13:16:05.028886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.028859 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tmp-dir" pod "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" (UID: "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:05.029111 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.029091 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.029197 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.029121 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nq29\" (UniqueName: \"kubernetes.io/projected/5c11b2df-1fc1-40ca-86f0-315f19e683e7-kube-api-access-4nq29\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.029197 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.029147 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.029197 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.029160 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c11b2df-1fc1-40ca-86f0-315f19e683e7-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.029197 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.029173 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.029197 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.029186 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf2h4\" (UniqueName: \"kubernetes.io/projected/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-kube-api-access-kf2h4\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.029452 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.029199 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.029452 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.029213 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c11b2df-1fc1-40ca-86f0-315f19e683e7-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.029452 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.029097 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-model-cache" pod "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" (UID: "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:05.031079 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.031053 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-dshm" pod "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" (UID: "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:05.031175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.031073 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tls-certs" pod "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" (UID: "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:16:05.031175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.031131 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-kube-api-access-lhrr6" pod "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" (UID: "a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2"). InnerVolumeSpecName "kube-api-access-lhrr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:05.073917 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.073888 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" Oct 31 13:16:05.073917 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.073911 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk" event={"ID":"ff621f22-5195-4616-9fb0-3604409740e7","Type":"ContainerDied","Data":"af58f701808c3be1a65123db21a155cdf4fc4df4849fab704b70dbf1e5765a74"} Oct 31 13:16:05.074181 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.073965 4542 scope.go:122] "RemoveContainer" containerID="2df4812e0f22adf1d00adb8d4f96a016ef6ac8c7e2c36464cdbdc8108c879d1a" Oct 31 13:16:05.076617 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.076445 4542 generic.go:356] "Generic (PLEG): container finished" podID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerID="5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a" exitCode=0 Oct 31 13:16:05.076617 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.076493 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" event={"ID":"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2","Type":"ContainerDied","Data":"5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a"} Oct 31 13:16:05.076617 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.076538 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" Oct 31 13:16:05.076617 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.076561 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" event={"ID":"a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2","Type":"ContainerDied","Data":"264827dad23a248356059e56df9968a2c90eb0cc35e6bde27953f46a04260e91"} Oct 31 13:16:05.078076 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.078050 4542 generic.go:356] "Generic (PLEG): container finished" podID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerID="42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2" exitCode=0 Oct 31 13:16:05.078201 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.078097 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" Oct 31 13:16:05.078201 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.078152 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" event={"ID":"5c11b2df-1fc1-40ca-86f0-315f19e683e7","Type":"ContainerDied","Data":"42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2"} Oct 31 13:16:05.078327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.078210 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" event={"ID":"5c11b2df-1fc1-40ca-86f0-315f19e683e7","Type":"ContainerDied","Data":"d5463b98a33a3799af07a158a57e50f4a45f231aaff1ede19830e1101f527aef"} Oct 31 13:16:05.079729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.079707 4542 generic.go:356] "Generic (PLEG): container finished" podID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerID="2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40" exitCode=0 Oct 31 13:16:05.079830 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.079752 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" Oct 31 13:16:05.079830 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.079794 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" event={"ID":"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937","Type":"ContainerDied","Data":"2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40"} Oct 31 13:16:05.079830 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.079820 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" event={"ID":"be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937","Type":"ContainerDied","Data":"08d1787182dbad2c92dc508f3d536e1d88bbac4f1969d5f83587386da33644b3"} Oct 31 13:16:05.085421 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.085402 4542 scope.go:122] "RemoveContainer" containerID="5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a" Oct 31 13:16:05.095491 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.095462 4542 scope.go:122] "RemoveContainer" containerID="5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a" Oct 31 13:16:05.096647 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:05.096621 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a\": container with ID starting with 5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a not found: ID does not exist" containerID="5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a" Oct 31 13:16:05.096752 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.096651 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a"} err="failed to get container status \"5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a\": rpc error: code = NotFound desc = could not find container \"5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a\": container with ID starting with 5967bb6945e1c3ecbaa67df973c928fcd89b049388d98e92a2a92cf9a0fa453a not found: ID does not exist" Oct 31 13:16:05.096752 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.096671 4542 scope.go:122] "RemoveContainer" containerID="42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2" Oct 31 13:16:05.104712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.104695 4542 scope.go:122] "RemoveContainer" containerID="42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2" Oct 31 13:16:05.105010 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:05.104993 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2\": container with ID starting with 42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2 not found: ID does not exist" containerID="42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2" Oct 31 13:16:05.105056 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.105017 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2"} err="failed to get container status \"42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2\": rpc error: code = NotFound desc = could not find container \"42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2\": container with ID starting with 42295969c26af4d00a993bd06f778f5c90bbc69267e3196e46595cbfb22385a2 not found: ID does not exist" Oct 31 13:16:05.105056 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.105029 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk"] Oct 31 13:16:05.105176 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.105032 4542 scope.go:122] "RemoveContainer" containerID="2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40" Oct 31 13:16:05.111417 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.111397 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-004f64-58fbfc7946-smdnk"] Oct 31 13:16:05.115146 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.115096 4542 scope.go:122] "RemoveContainer" containerID="2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40" Oct 31 13:16:05.115364 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:05.115348 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40\": container with ID starting with 2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40 not found: ID does not exist" containerID="2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40" Oct 31 13:16:05.115427 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.115377 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40"} err="failed to get container status \"2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40\": rpc error: code = NotFound desc = could not find container \"2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40\": container with ID starting with 2aeab03fb49cbd283f2013b7ce0314e4a6678b0f98f29d9ecafcf7d248d6ea40 not found: ID does not exist" Oct 31 13:16:05.126703 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.126685 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn"] Oct 31 13:16:05.129053 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.129032 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn"] Oct 31 13:16:05.130096 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.130072 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.130169 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.130099 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrr6\" (UniqueName: \"kubernetes.io/projected/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-kube-api-access-lhrr6\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.130169 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.130116 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.130169 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.130129 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:05.140787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.140767 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm"] Oct 31 13:16:05.145853 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.145835 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm"] Oct 31 13:16:05.160116 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.160090 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk"] Oct 31 13:16:05.163492 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.163456 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk"] Oct 31 13:16:05.291790 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.291760 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" path="/var/lib/kubelet/pods/5c11b2df-1fc1-40ca-86f0-315f19e683e7/volumes" Oct 31 13:16:05.292213 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.292201 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" path="/var/lib/kubelet/pods/a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2/volumes" Oct 31 13:16:05.292631 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.292615 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" path="/var/lib/kubelet/pods/be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937/volumes" Oct 31 13:16:05.293301 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.293287 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3f742af-293a-40d6-9a93-75e3030306b8" path="/var/lib/kubelet/pods/c3f742af-293a-40d6-9a93-75e3030306b8/volumes" Oct 31 13:16:05.293711 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.293698 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff621f22-5195-4616-9fb0-3604409740e7" path="/var/lib/kubelet/pods/ff621f22-5195-4616-9fb0-3604409740e7/volumes" Oct 31 13:16:05.788011 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.787973 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-w0-7bf66c-kserve-6f658ffc67-fjfpm" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:16:05.819334 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.819295 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/distinct-2-w0-7bf66c-kserve-788b84755b-6spwk" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:16:05.833612 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.833579 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw"] Oct 31 13:16:05.834267 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834248 4542 cpu_manager.go:414] "containerMap: removing container" podUID="c3f742af-293a-40d6-9a93-75e3030306b8" containerName="payload-pre-processing" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834269 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f742af-293a-40d6-9a93-75e3030306b8" containerName="payload-pre-processing" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834285 4542 cpu_manager.go:414] "containerMap: removing container" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834291 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834301 4542 cpu_manager.go:414] "containerMap: removing container" podUID="2541cf2f-8571-49cc-995f-4174130110be" containerName="maas-api" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834306 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2541cf2f-8571-49cc-995f-4174130110be" containerName="maas-api" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834317 4542 cpu_manager.go:414] "containerMap: removing container" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834322 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834332 4542 cpu_manager.go:414] "containerMap: removing container" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834336 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834349 4542 cpu_manager.go:414] "containerMap: removing container" podUID="ff621f22-5195-4616-9fb0-3604409740e7" containerName="maas-api" Oct 31 13:16:05.834351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834354 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff621f22-5195-4616-9fb0-3604409740e7" containerName="maas-api" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834359 4542 cpu_manager.go:414] "containerMap: removing container" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834364 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834373 4542 cpu_manager.go:414] "containerMap: removing container" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834377 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834393 4542 cpu_manager.go:414] "containerMap: removing container" podUID="6d9922f9-afbe-4381-85a8-7c77c251cfba" containerName="revoke-keys" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834397 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9922f9-afbe-4381-85a8-7c77c251cfba" containerName="revoke-keys" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834404 4542 cpu_manager.go:414] "containerMap: removing container" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834408 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834414 4542 cpu_manager.go:414] "containerMap: removing container" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834419 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834544 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f742af-293a-40d6-9a93-75e3030306b8" containerName="payload-pre-processing" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834559 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6be33c-8d58-4ebf-9f5c-e1e4a6b2b937" containerName="main" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834608 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="2541cf2f-8571-49cc-995f-4174130110be" containerName="maas-api" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834622 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834632 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3ae33d-ed8c-4ce2-ba24-53be34159ba1" containerName="main" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834641 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff621f22-5195-4616-9fb0-3604409740e7" containerName="maas-api" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834646 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834652 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d5e01c-7a06-4122-b5d9-0e76ff7b0de2" containerName="main" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834660 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9922f9-afbe-4381-85a8-7c77c251cfba" containerName="revoke-keys" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834666 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68e7546-4bb1-4dd6-8f71-2d3c82c9e9c0" containerName="istio-proxy" Oct 31 13:16:05.834729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.834672 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7c3b8b-6f23-4fe9-be0e-42b98613bcee" containerName="istio-proxy" Oct 31 13:16:05.835689 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.835659 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/premium-w0-7bf66c-kserve-78c757bc48-4rfsn" podUID="5c11b2df-1fc1-40ca-86f0-315f19e683e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 13:16:05.839271 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.839254 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:05.844788 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.844749 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 13:16:05.844871 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.844784 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kc68z\"" Oct 31 13:16:05.844871 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.844863 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 13:16:05.847492 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.847461 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 13:16:05.852712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.852690 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw"] Oct 31 13:16:05.940555 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.940528 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lq2p\" (UniqueName: \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-kube-api-access-2lq2p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:05.940753 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.940727 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:05.940883 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.940760 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:05.940883 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.940777 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:05.940883 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.940834 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae57b233-eccd-45cf-a8ec-93f681c4c042-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:05.940982 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.940908 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:05.940982 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:05.940927 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.042026 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.041947 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.042026 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.041991 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.042218 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.042170 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lq2p\" (UniqueName: \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-kube-api-access-2lq2p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.042337 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.042323 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.042413 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.042347 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.042413 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.042363 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.042413 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.042383 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae57b233-eccd-45cf-a8ec-93f681c4c042-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.042706 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.042681 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.044774 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.044747 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.044774 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.044763 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.044947 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.044826 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae57b233-eccd-45cf-a8ec-93f681c4c042-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.045027 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.045009 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.059194 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.059170 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.059486 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.059451 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lq2p\" (UniqueName: \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-kube-api-access-2lq2p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jqfzw\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.152423 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.152391 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:06.439072 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.439047 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw"] Oct 31 13:16:06.440730 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:16:06.440703 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae57b233_eccd_45cf_a8ec_93f681c4c042.slice/crio-3d53b00f21128adb0f5c818512c3f99f15832072bd76b3a078eb0410d27d2e64 WatchSource:0}: Error finding container 3d53b00f21128adb0f5c818512c3f99f15832072bd76b3a078eb0410d27d2e64: Status 404 returned error can't find the container with id 3d53b00f21128adb0f5c818512c3f99f15832072bd76b3a078eb0410d27d2e64 Oct 31 13:16:06.443354 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:06.443322 4542 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:16:07.097371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:07.097334 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" event={"ID":"ae57b233-eccd-45cf-a8ec-93f681c4c042","Type":"ContainerStarted","Data":"3d53b00f21128adb0f5c818512c3f99f15832072bd76b3a078eb0410d27d2e64"} Oct 31 13:16:07.956956 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:07.956901 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67"] Oct 31 13:16:07.957386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:07.957150 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" containerID="cri-o://a79da1681beb6c1c6783c437edde99b3867dce095d95539245c5accca67aae3a" gracePeriod=30 Oct 31 13:16:07.974224 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:07.974188 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn"] Oct 31 13:16:07.974721 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:07.974689 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" containerID="cri-o://b2016364e14ce472d4cc9c0007791ccaf0bb3a4a81338886324bb6416fea4620" gracePeriod=30 Oct 31 13:16:07.984702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:07.984680 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv"] Oct 31 13:16:07.984868 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:07.984850 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" containerID="cri-o://052a9350b364d89b9bd5b10c3f565122db1d1d0f0239d00d69c450b7a8689f3f" gracePeriod=30 Oct 31 13:16:07.987332 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:07.987305 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw"] Oct 31 13:16:07.988013 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:07.987520 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" containerID="cri-o://59e673e9746826f22d78396ca06993de0da31eaa0ce54c897e25aaa03783246d" gracePeriod=30 Oct 31 13:16:08.212288 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:08.212199 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:08.212288 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:08.212254 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:08.212533 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:08.212303 4542 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:08.212533 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:08.212330 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:09.177500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:09.173581 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf"] Oct 31 13:16:09.177500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:09.174441 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" containerID="cri-o://283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4" gracePeriod=30 Oct 31 13:16:09.177500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:09.175642 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht"] Oct 31 13:16:09.177500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:09.175950 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" containerID="cri-o://b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645" gracePeriod=30 Oct 31 13:16:09.222577 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:09.185563 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs"] Oct 31 13:16:09.222577 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:09.185769 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerName="istio-proxy" containerID="cri-o://b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86" gracePeriod=30 Oct 31 13:16:09.236205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:09.233986 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs"] Oct 31 13:16:09.236205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:09.234260 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" containerID="cri-o://d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21" gracePeriod=30 Oct 31 13:16:12.069043 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:12.069003 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5"] Oct 31 13:16:12.069562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:12.069265 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" containerID="cri-o://103b758424a7ec7fd142f8bff3a018b48d84fdef3ff7a5af3c11459abd0e860a" gracePeriod=30 Oct 31 13:16:12.072050 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:12.071978 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb"] Oct 31 13:16:12.072317 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:12.072273 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" containerID="cri-o://b1127fccc9168a3056055ebb0fee7b34e48ff64e03e23eba1cea601d572fdab6" gracePeriod=30 Oct 31 13:16:12.074905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:12.074881 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c"] Oct 31 13:16:12.075092 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:12.075069 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" containerID="cri-o://0af2b04ccc271ab6f25bf533782c5535b14cb83381f42aa675437ce156786c87" gracePeriod=30 Oct 31 13:16:12.077356 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:12.077333 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6"] Oct 31 13:16:12.077542 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:12.077511 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" containerID="cri-o://97ede513ae6cfd2cae07d632c12691bd4b52079aae5e43ca4ac55257408be486" gracePeriod=30 Oct 31 13:16:13.073229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.073201 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf"] Oct 31 13:16:13.073621 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.073413 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" containerID="cri-o://39a0dcaf7e4ce535078f9c07bb52ceb47cc891394c50ece4404e70bcceb98b2b" gracePeriod=30 Oct 31 13:16:13.076041 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.076006 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq"] Oct 31 13:16:13.076205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.076181 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" containerID="cri-o://b48ebd6102c9beb09791a08c40b5fd062244248748fcf5d514de4ea0671637ed" gracePeriod=30 Oct 31 13:16:13.079398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.079372 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r"] Oct 31 13:16:13.079711 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.079688 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" containerID="cri-o://7e6c4dab7f1c7226a882c66666f84a522da59b8ff7c77fc92cc4386b627b7bf4" gracePeriod=30 Oct 31 13:16:13.089683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.089602 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln"] Oct 31 13:16:13.089777 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.089757 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerName="istio-proxy" containerID="cri-o://2b531afbb82da126495810f2dc90b77be0c542ea789237ad4b4b11d3943eda13" gracePeriod=30 Oct 31 13:16:13.147692 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.147176 4542 generic.go:356] "Generic (PLEG): container finished" podID="32ddbe95-4c3e-494d-9257-accc94089b63" containerID="b2016364e14ce472d4cc9c0007791ccaf0bb3a4a81338886324bb6416fea4620" exitCode=0 Oct 31 13:16:13.147692 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.147282 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" event={"ID":"32ddbe95-4c3e-494d-9257-accc94089b63","Type":"ContainerDied","Data":"b2016364e14ce472d4cc9c0007791ccaf0bb3a4a81338886324bb6416fea4620"} Oct 31 13:16:13.150155 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.149923 4542 generic.go:356] "Generic (PLEG): container finished" podID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerID="59e673e9746826f22d78396ca06993de0da31eaa0ce54c897e25aaa03783246d" exitCode=0 Oct 31 13:16:13.150155 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.150040 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" event={"ID":"eabf6be0-70ac-4a23-bce4-23a0047c0cfc","Type":"ContainerDied","Data":"59e673e9746826f22d78396ca06993de0da31eaa0ce54c897e25aaa03783246d"} Oct 31 13:16:13.152404 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.152213 4542 generic.go:356] "Generic (PLEG): container finished" podID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerID="a79da1681beb6c1c6783c437edde99b3867dce095d95539245c5accca67aae3a" exitCode=0 Oct 31 13:16:13.152404 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.152342 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" event={"ID":"2f8e330f-01c1-435e-b684-56a1ed9208b7","Type":"ContainerDied","Data":"a79da1681beb6c1c6783c437edde99b3867dce095d95539245c5accca67aae3a"} Oct 31 13:16:13.154446 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.154306 4542 generic.go:356] "Generic (PLEG): container finished" podID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerID="052a9350b364d89b9bd5b10c3f565122db1d1d0f0239d00d69c450b7a8689f3f" exitCode=0 Oct 31 13:16:13.154446 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.154357 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" event={"ID":"f067868a-a184-4ef4-9f5c-7de384be46ae","Type":"ContainerDied","Data":"052a9350b364d89b9bd5b10c3f565122db1d1d0f0239d00d69c450b7a8689f3f"} Oct 31 13:16:13.317906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.317873 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:16:13.346976 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.346951 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-certs\") pod \"f067868a-a184-4ef4-9f5c-7de384be46ae\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " Oct 31 13:16:13.347067 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.347005 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-kube-api-access-75s6v\" (UniqueName: \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-kube-api-access-75s6v\") pod \"f067868a-a184-4ef4-9f5c-7de384be46ae\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " Oct 31 13:16:13.347067 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.347050 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-token\" (UniqueName: \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-token\") pod \"f067868a-a184-4ef4-9f5c-7de384be46ae\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " Oct 31 13:16:13.347172 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.347095 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-data\") pod \"f067868a-a184-4ef4-9f5c-7de384be46ae\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " Oct 31 13:16:13.347172 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.347157 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-envoy\") pod \"f067868a-a184-4ef4-9f5c-7de384be46ae\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " Oct 31 13:16:13.347284 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.347271 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-socket\") pod \"f067868a-a184-4ef4-9f5c-7de384be46ae\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " Oct 31 13:16:13.347336 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.347304 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-credential-socket\") pod \"f067868a-a184-4ef4-9f5c-7de384be46ae\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " Oct 31 13:16:13.347389 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.347354 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f067868a-a184-4ef4-9f5c-7de384be46ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f067868a-a184-4ef4-9f5c-7de384be46ae-istiod-ca-cert\") pod \"f067868a-a184-4ef4-9f5c-7de384be46ae\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " Oct 31 13:16:13.347717 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.347693 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-podinfo\") pod \"f067868a-a184-4ef4-9f5c-7de384be46ae\" (UID: \"f067868a-a184-4ef4-9f5c-7de384be46ae\") " Oct 31 13:16:13.348015 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.347987 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-socket" pod "f067868a-a184-4ef4-9f5c-7de384be46ae" (UID: "f067868a-a184-4ef4-9f5c-7de384be46ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.348132 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.348009 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-credential-socket" pod "f067868a-a184-4ef4-9f5c-7de384be46ae" (UID: "f067868a-a184-4ef4-9f5c-7de384be46ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.348239 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.348209 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-certs" pod "f067868a-a184-4ef4-9f5c-7de384be46ae" (UID: "f067868a-a184-4ef4-9f5c-7de384be46ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.348341 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.348316 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-data" pod "f067868a-a184-4ef4-9f5c-7de384be46ae" (UID: "f067868a-a184-4ef4-9f5c-7de384be46ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.348961 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.348721 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f067868a-a184-4ef4-9f5c-7de384be46ae-istiod-ca-cert" pod "f067868a-a184-4ef4-9f5c-7de384be46ae" (UID: "f067868a-a184-4ef4-9f5c-7de384be46ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:13.348961 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.348730 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.348961 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.348765 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.348961 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.348783 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.348961 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.348798 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.350886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.350858 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-token" pod "f067868a-a184-4ef4-9f5c-7de384be46ae" (UID: "f067868a-a184-4ef4-9f5c-7de384be46ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:13.353903 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.353862 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-podinfo" pod "f067868a-a184-4ef4-9f5c-7de384be46ae" (UID: "f067868a-a184-4ef4-9f5c-7de384be46ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:13.354152 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.354118 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-kube-api-access-75s6v" pod "f067868a-a184-4ef4-9f5c-7de384be46ae" (UID: "f067868a-a184-4ef4-9f5c-7de384be46ae"). InnerVolumeSpecName "kube-api-access-75s6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:13.357223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.357190 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-envoy" pod "f067868a-a184-4ef4-9f5c-7de384be46ae" (UID: "f067868a-a184-4ef4-9f5c-7de384be46ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.364366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.364344 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:16:13.367235 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.367162 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:16:13.442528 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.437364 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:16:13.450121 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450085 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-token\" (UniqueName: \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-token\") pod \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " Oct 31 13:16:13.450263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450145 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-certs\") pod \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " Oct 31 13:16:13.450263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450191 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-socket\") pod \"32ddbe95-4c3e-494d-9257-accc94089b63\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " Oct 31 13:16:13.450263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450233 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istiod-ca-cert\") pod \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " Oct 31 13:16:13.450419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450278 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-kube-api-access-p7kbx\" (UniqueName: \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-kube-api-access-p7kbx\") pod \"2f8e330f-01c1-435e-b684-56a1ed9208b7\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " Oct 31 13:16:13.450419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450319 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-socket\") pod \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " Oct 31 13:16:13.450419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450347 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8e330f-01c1-435e-b684-56a1ed9208b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8e330f-01c1-435e-b684-56a1ed9208b7-istiod-ca-cert\") pod \"2f8e330f-01c1-435e-b684-56a1ed9208b7\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " Oct 31 13:16:13.450419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450375 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-istio-token\" (UniqueName: \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-istio-token\") pod \"32ddbe95-4c3e-494d-9257-accc94089b63\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " Oct 31 13:16:13.450419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450402 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-kube-api-access-xg67m\" (UniqueName: \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-kube-api-access-xg67m\") pod \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " Oct 31 13:16:13.450685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450447 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-kube-api-access-cpvc6\" (UniqueName: \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-kube-api-access-cpvc6\") pod \"32ddbe95-4c3e-494d-9257-accc94089b63\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " Oct 31 13:16:13.450685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450501 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-certs\") pod \"32ddbe95-4c3e-494d-9257-accc94089b63\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " Oct 31 13:16:13.450685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450547 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-credential-socket\") pod \"32ddbe95-4c3e-494d-9257-accc94089b63\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " Oct 31 13:16:13.450685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450590 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-podinfo\") pod \"2f8e330f-01c1-435e-b684-56a1ed9208b7\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " Oct 31 13:16:13.450685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450618 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-data\") pod \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " Oct 31 13:16:13.450685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450666 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-envoy\") pod \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " Oct 31 13:16:13.450989 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450694 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-credential-socket\") pod \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " Oct 31 13:16:13.450989 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450755 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-certs" pod "eabf6be0-70ac-4a23-bce4-23a0047c0cfc" (UID: "eabf6be0-70ac-4a23-bce4-23a0047c0cfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.450989 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450765 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-envoy\") pod \"2f8e330f-01c1-435e-b684-56a1ed9208b7\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " Oct 31 13:16:13.450989 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450823 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-data\") pod \"32ddbe95-4c3e-494d-9257-accc94089b63\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " Oct 31 13:16:13.450989 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450945 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-envoy\") pod \"32ddbe95-4c3e-494d-9257-accc94089b63\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " Oct 31 13:16:13.451229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.450996 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-podinfo\") pod \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\" (UID: \"eabf6be0-70ac-4a23-bce4-23a0047c0cfc\") " Oct 31 13:16:13.451229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.451055 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-token\" (UniqueName: \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-token\") pod \"2f8e330f-01c1-435e-b684-56a1ed9208b7\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " Oct 31 13:16:13.451229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.451094 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-data\") pod \"2f8e330f-01c1-435e-b684-56a1ed9208b7\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " Oct 31 13:16:13.451229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.451126 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ddbe95-4c3e-494d-9257-accc94089b63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddbe95-4c3e-494d-9257-accc94089b63-istiod-ca-cert\") pod \"32ddbe95-4c3e-494d-9257-accc94089b63\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " Oct 31 13:16:13.451229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.451187 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-certs\") pod \"2f8e330f-01c1-435e-b684-56a1ed9208b7\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " Oct 31 13:16:13.451229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.451216 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32ddbe95-4c3e-494d-9257-accc94089b63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddbe95-4c3e-494d-9257-accc94089b63-istio-podinfo\") pod \"32ddbe95-4c3e-494d-9257-accc94089b63\" (UID: \"32ddbe95-4c3e-494d-9257-accc94089b63\") " Oct 31 13:16:13.451530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.451259 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-credential-socket\") pod \"2f8e330f-01c1-435e-b684-56a1ed9208b7\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " Oct 31 13:16:13.451530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.451295 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-socket\") pod \"2f8e330f-01c1-435e-b684-56a1ed9208b7\" (UID: \"2f8e330f-01c1-435e-b684-56a1ed9208b7\") " Oct 31 13:16:13.451810 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.451772 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-data" pod "2f8e330f-01c1-435e-b684-56a1ed9208b7" (UID: "2f8e330f-01c1-435e-b684-56a1ed9208b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.451992 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.451971 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-socket" pod "eabf6be0-70ac-4a23-bce4-23a0047c0cfc" (UID: "eabf6be0-70ac-4a23-bce4-23a0047c0cfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.452198 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.452164 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-data" pod "32ddbe95-4c3e-494d-9257-accc94089b63" (UID: "32ddbe95-4c3e-494d-9257-accc94089b63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.453024 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.452600 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istiod-ca-cert" pod "eabf6be0-70ac-4a23-bce4-23a0047c0cfc" (UID: "eabf6be0-70ac-4a23-bce4-23a0047c0cfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:13.453024 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.452987 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-certs" pod "32ddbe95-4c3e-494d-9257-accc94089b63" (UID: "32ddbe95-4c3e-494d-9257-accc94089b63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.453501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.453290 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8e330f-01c1-435e-b684-56a1ed9208b7-istiod-ca-cert" pod "2f8e330f-01c1-435e-b684-56a1ed9208b7" (UID: "2f8e330f-01c1-435e-b684-56a1ed9208b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:13.454831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.453653 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-credential-socket" pod "32ddbe95-4c3e-494d-9257-accc94089b63" (UID: "32ddbe95-4c3e-494d-9257-accc94089b63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.454831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.453704 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-envoy" pod "2f8e330f-01c1-435e-b684-56a1ed9208b7" (UID: "2f8e330f-01c1-435e-b684-56a1ed9208b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.454831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.454119 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-data" pod "eabf6be0-70ac-4a23-bce4-23a0047c0cfc" (UID: "eabf6be0-70ac-4a23-bce4-23a0047c0cfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.454831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.454154 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-certs" pod "2f8e330f-01c1-435e-b684-56a1ed9208b7" (UID: "2f8e330f-01c1-435e-b684-56a1ed9208b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.454831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.454282 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-token" pod "eabf6be0-70ac-4a23-bce4-23a0047c0cfc" (UID: "eabf6be0-70ac-4a23-bce4-23a0047c0cfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:13.454831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.454346 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-credential-socket" pod "eabf6be0-70ac-4a23-bce4-23a0047c0cfc" (UID: "eabf6be0-70ac-4a23-bce4-23a0047c0cfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.454831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.454366 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ddbe95-4c3e-494d-9257-accc94089b63-istiod-ca-cert" pod "32ddbe95-4c3e-494d-9257-accc94089b63" (UID: "32ddbe95-4c3e-494d-9257-accc94089b63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:13.454831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.454451 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-credential-socket" pod "2f8e330f-01c1-435e-b684-56a1ed9208b7" (UID: "2f8e330f-01c1-435e-b684-56a1ed9208b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.454831 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.454617 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-socket" pod "2f8e330f-01c1-435e-b684-56a1ed9208b7" (UID: "2f8e330f-01c1-435e-b684-56a1ed9208b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.455318 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.454975 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-envoy" pod "32ddbe95-4c3e-494d-9257-accc94089b63" (UID: "32ddbe95-4c3e-494d-9257-accc94089b63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.455775 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.455746 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-socket" pod "32ddbe95-4c3e-494d-9257-accc94089b63" (UID: "32ddbe95-4c3e-494d-9257-accc94089b63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.456690 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.456669 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.456861 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.456845 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.456982 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.456967 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.457143 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.457120 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.457249 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.457235 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75s6v\" (UniqueName: \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-kube-api-access-75s6v\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.457333 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.457322 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.457411 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.457399 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.457594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.457579 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddbe95-4c3e-494d-9257-accc94089b63-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.457690 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.457678 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.457790 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.457777 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.457872 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.457859 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8e330f-01c1-435e-b684-56a1ed9208b7-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.457961 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.457950 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f067868a-a184-4ef4-9f5c-7de384be46ae-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458079 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458064 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458164 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458152 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458258 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458246 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458328 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458435 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458422 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f067868a-a184-4ef4-9f5c-7de384be46ae-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458547 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458531 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458630 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458619 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8e330f-01c1-435e-b684-56a1ed9208b7-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458711 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458700 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458832 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458809 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddbe95-4c3e-494d-9257-accc94089b63-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458905 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.458996 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.458985 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.459488 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.459440 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-kube-api-access-xg67m" pod "eabf6be0-70ac-4a23-bce4-23a0047c0cfc" (UID: "eabf6be0-70ac-4a23-bce4-23a0047c0cfc"). InnerVolumeSpecName "kube-api-access-xg67m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:13.459750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.459676 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-token" pod "2f8e330f-01c1-435e-b684-56a1ed9208b7" (UID: "2f8e330f-01c1-435e-b684-56a1ed9208b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:13.459750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.459721 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-envoy" pod "eabf6be0-70ac-4a23-bce4-23a0047c0cfc" (UID: "eabf6be0-70ac-4a23-bce4-23a0047c0cfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:13.460003 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.459979 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32ddbe95-4c3e-494d-9257-accc94089b63-istio-podinfo" pod "32ddbe95-4c3e-494d-9257-accc94089b63" (UID: "32ddbe95-4c3e-494d-9257-accc94089b63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:13.460909 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.460885 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-podinfo" pod "eabf6be0-70ac-4a23-bce4-23a0047c0cfc" (UID: "eabf6be0-70ac-4a23-bce4-23a0047c0cfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:13.461178 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.461146 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-kube-api-access-p7kbx" pod "2f8e330f-01c1-435e-b684-56a1ed9208b7" (UID: "2f8e330f-01c1-435e-b684-56a1ed9208b7"). InnerVolumeSpecName "kube-api-access-p7kbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:13.461245 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.461210 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-istio-token" pod "32ddbe95-4c3e-494d-9257-accc94089b63" (UID: "32ddbe95-4c3e-494d-9257-accc94089b63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:13.461771 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.461747 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-kube-api-access-cpvc6" pod "32ddbe95-4c3e-494d-9257-accc94089b63" (UID: "32ddbe95-4c3e-494d-9257-accc94089b63"). InnerVolumeSpecName "kube-api-access-cpvc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:13.463646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.463623 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-podinfo" pod "2f8e330f-01c1-435e-b684-56a1ed9208b7" (UID: "2f8e330f-01c1-435e-b684-56a1ed9208b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:13.560182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.560102 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpvc6\" (UniqueName: \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-kube-api-access-cpvc6\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.560182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.560131 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.560182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.560143 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.560182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.560153 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.560182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.560161 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.560182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.560172 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddbe95-4c3e-494d-9257-accc94089b63-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.560182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.560185 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7kbx\" (UniqueName: \"kubernetes.io/projected/2f8e330f-01c1-435e-b684-56a1ed9208b7-kube-api-access-p7kbx\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.560516 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.560194 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddbe95-4c3e-494d-9257-accc94089b63-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:13.560516 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:13.560202 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg67m\" (UniqueName: \"kubernetes.io/projected/eabf6be0-70ac-4a23-bce4-23a0047c0cfc-kube-api-access-xg67m\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.162279 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.162243 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" Oct 31 13:16:14.162734 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.162242 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn" event={"ID":"32ddbe95-4c3e-494d-9257-accc94089b63","Type":"ContainerDied","Data":"4a2b90fb4c948b1cad1bcd7d48721dce2a5b69f276c6f5334093ea5f46efd60f"} Oct 31 13:16:14.162734 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.162398 4542 scope.go:122] "RemoveContainer" containerID="b2016364e14ce472d4cc9c0007791ccaf0bb3a4a81338886324bb6416fea4620" Oct 31 13:16:14.164030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.163957 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" Oct 31 13:16:14.164030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.163979 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw" event={"ID":"eabf6be0-70ac-4a23-bce4-23a0047c0cfc","Type":"ContainerDied","Data":"25cb2e4d0248f06bd82effb470cc81f252c675bcf692a1fa713a124e0f139d37"} Oct 31 13:16:14.172987 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.165631 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" Oct 31 13:16:14.172987 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.165659 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67" event={"ID":"2f8e330f-01c1-435e-b684-56a1ed9208b7","Type":"ContainerDied","Data":"1554f90fec441436043b0b2c314b4179e4cc2d3f2e37043399d2d80ad175526a"} Oct 31 13:16:14.176691 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.176548 4542 scope.go:122] "RemoveContainer" containerID="59e673e9746826f22d78396ca06993de0da31eaa0ce54c897e25aaa03783246d" Oct 31 13:16:14.177329 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.177306 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" event={"ID":"f067868a-a184-4ef4-9f5c-7de384be46ae","Type":"ContainerDied","Data":"f339e3ea00dbac319a0a912ad092b8bec274c0abc98a490bba15d62a53ca571a"} Oct 31 13:16:14.177421 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.177394 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv" Oct 31 13:16:14.189902 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.189882 4542 scope.go:122] "RemoveContainer" containerID="a79da1681beb6c1c6783c437edde99b3867dce095d95539245c5accca67aae3a" Oct 31 13:16:14.199953 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.199838 4542 scope.go:122] "RemoveContainer" containerID="052a9350b364d89b9bd5b10c3f565122db1d1d0f0239d00d69c450b7a8689f3f" Oct 31 13:16:14.219457 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.219430 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn"] Oct 31 13:16:14.223419 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.223390 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gjjkn"] Oct 31 13:16:14.229414 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.229389 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw"] Oct 31 13:16:14.237024 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.236970 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5xvnw"] Oct 31 13:16:14.249530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.249502 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv"] Oct 31 13:16:14.256830 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.256805 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-gr4mv"] Oct 31 13:16:14.270784 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.270753 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67"] Oct 31 13:16:14.273394 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.273321 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:14.273524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.273391 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:14.274378 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.274357 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-5jz67"] Oct 31 13:16:14.297675 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.297646 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:14.297810 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.297694 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:14.487727 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.487580 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:16:14.489702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.489680 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.575582 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-data\") pod \"2d057adb-f72c-4327-b979-aa61fdf98068\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.575681 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-certs\") pod \"5c7eb1ab-7542-44aa-a93d-d3186844631a\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.575875 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-data" pod "2d057adb-f72c-4327-b979-aa61fdf98068" (UID: "2d057adb-f72c-4327-b979-aa61fdf98068"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.575976 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-certs" pod "5c7eb1ab-7542-44aa-a93d-d3186844631a" (UID: "5c7eb1ab-7542-44aa-a93d-d3186844631a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.576224 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-socket" pod "5c7eb1ab-7542-44aa-a93d-d3186844631a" (UID: "5c7eb1ab-7542-44aa-a93d-d3186844631a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.576353 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-socket\") pod \"5c7eb1ab-7542-44aa-a93d-d3186844631a\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.576413 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-certs\") pod \"2d057adb-f72c-4327-b979-aa61fdf98068\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.576688 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-certs" pod "2d057adb-f72c-4327-b979-aa61fdf98068" (UID: "2d057adb-f72c-4327-b979-aa61fdf98068"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.576752 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-token\" (UniqueName: \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-token\") pod \"5c7eb1ab-7542-44aa-a93d-d3186844631a\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.577083 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-envoy\") pod \"5c7eb1ab-7542-44aa-a93d-d3186844631a\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.577135 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-envoy\") pod \"2d057adb-f72c-4327-b979-aa61fdf98068\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.577166 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-podinfo\") pod \"5c7eb1ab-7542-44aa-a93d-d3186844631a\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " Oct 31 13:16:14.577514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.577213 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-credential-socket\") pod \"5c7eb1ab-7542-44aa-a93d-d3186844631a\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " Oct 31 13:16:14.579398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.577733 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-credential-socket\") pod \"2d057adb-f72c-4327-b979-aa61fdf98068\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " Oct 31 13:16:14.579398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.577788 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7eb1ab-7542-44aa-a93d-d3186844631a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7eb1ab-7542-44aa-a93d-d3186844631a-istiod-ca-cert\") pod \"5c7eb1ab-7542-44aa-a93d-d3186844631a\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " Oct 31 13:16:14.579398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.577835 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d057adb-f72c-4327-b979-aa61fdf98068-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d057adb-f72c-4327-b979-aa61fdf98068-istiod-ca-cert\") pod \"2d057adb-f72c-4327-b979-aa61fdf98068\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " Oct 31 13:16:14.579398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.577872 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d057adb-f72c-4327-b979-aa61fdf98068-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d057adb-f72c-4327-b979-aa61fdf98068-istio-podinfo\") pod \"2d057adb-f72c-4327-b979-aa61fdf98068\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " Oct 31 13:16:14.579398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.577986 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-istio-token\" (UniqueName: \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-istio-token\") pod \"2d057adb-f72c-4327-b979-aa61fdf98068\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " Oct 31 13:16:14.579398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.578021 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-kube-api-access-pr6nf\" (UniqueName: \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-kube-api-access-pr6nf\") pod \"2d057adb-f72c-4327-b979-aa61fdf98068\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " Oct 31 13:16:14.579398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.578066 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-data\") pod \"5c7eb1ab-7542-44aa-a93d-d3186844631a\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " Oct 31 13:16:14.579398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.578114 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-socket\") pod \"2d057adb-f72c-4327-b979-aa61fdf98068\" (UID: \"2d057adb-f72c-4327-b979-aa61fdf98068\") " Oct 31 13:16:14.579398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.578192 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-kube-api-access-2w5jl\" (UniqueName: \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-kube-api-access-2w5jl\") pod \"5c7eb1ab-7542-44aa-a93d-d3186844631a\" (UID: \"5c7eb1ab-7542-44aa-a93d-d3186844631a\") " Oct 31 13:16:14.579398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.579139 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d057adb-f72c-4327-b979-aa61fdf98068-istiod-ca-cert" pod "2d057adb-f72c-4327-b979-aa61fdf98068" (UID: "2d057adb-f72c-4327-b979-aa61fdf98068"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:14.579905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.579726 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.579905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.579751 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.579905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.579766 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.579905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.579779 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.579905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.579794 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d057adb-f72c-4327-b979-aa61fdf98068-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.580153 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.580080 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-data" pod "5c7eb1ab-7542-44aa-a93d-d3186844631a" (UID: "5c7eb1ab-7542-44aa-a93d-d3186844631a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.580283 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.580260 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-socket" pod "2d057adb-f72c-4327-b979-aa61fdf98068" (UID: "2d057adb-f72c-4327-b979-aa61fdf98068"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.581763 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.581532 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d057adb-f72c-4327-b979-aa61fdf98068-istio-podinfo" pod "2d057adb-f72c-4327-b979-aa61fdf98068" (UID: "2d057adb-f72c-4327-b979-aa61fdf98068"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:14.583101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.582939 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-envoy" pod "2d057adb-f72c-4327-b979-aa61fdf98068" (UID: "2d057adb-f72c-4327-b979-aa61fdf98068"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.583101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.582946 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-token" pod "5c7eb1ab-7542-44aa-a93d-d3186844631a" (UID: "5c7eb1ab-7542-44aa-a93d-d3186844631a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:14.583445 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.583422 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-credential-socket" pod "2d057adb-f72c-4327-b979-aa61fdf98068" (UID: "2d057adb-f72c-4327-b979-aa61fdf98068"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.583695 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.583673 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-credential-socket" pod "5c7eb1ab-7542-44aa-a93d-d3186844631a" (UID: "5c7eb1ab-7542-44aa-a93d-d3186844631a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.583749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.583726 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7eb1ab-7542-44aa-a93d-d3186844631a-istiod-ca-cert" pod "5c7eb1ab-7542-44aa-a93d-d3186844631a" (UID: "5c7eb1ab-7542-44aa-a93d-d3186844631a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:14.584916 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.584883 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-kube-api-access-2w5jl" pod "5c7eb1ab-7542-44aa-a93d-d3186844631a" (UID: "5c7eb1ab-7542-44aa-a93d-d3186844631a"). InnerVolumeSpecName "kube-api-access-2w5jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:14.585020 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.584956 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-podinfo" pod "5c7eb1ab-7542-44aa-a93d-d3186844631a" (UID: "5c7eb1ab-7542-44aa-a93d-d3186844631a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:14.586075 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.586048 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-kube-api-access-pr6nf" pod "2d057adb-f72c-4327-b979-aa61fdf98068" (UID: "2d057adb-f72c-4327-b979-aa61fdf98068"). InnerVolumeSpecName "kube-api-access-pr6nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:14.586897 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.586877 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-envoy" pod "5c7eb1ab-7542-44aa-a93d-d3186844631a" (UID: "5c7eb1ab-7542-44aa-a93d-d3186844631a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.587092 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.587077 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-istio-token" pod "2d057adb-f72c-4327-b979-aa61fdf98068" (UID: "2d057adb-f72c-4327-b979-aa61fdf98068"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:14.608170 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.608150 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:16:14.620445 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.620419 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:16:14.680675 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.680644 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-certs\") pod \"2604e553-ae64-4fdf-ba10-1a639e5157f0\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " Oct 31 13:16:14.680864 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.680687 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-podinfo\") pod \"2604e553-ae64-4fdf-ba10-1a639e5157f0\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " Oct 31 13:16:14.680864 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.680743 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-data\") pod \"2604e553-ae64-4fdf-ba10-1a639e5157f0\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " Oct 31 13:16:14.680864 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.680788 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-envoy\") pod \"2604e553-ae64-4fdf-ba10-1a639e5157f0\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " Oct 31 13:16:14.681067 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.680931 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-certs" pod "2604e553-ae64-4fdf-ba10-1a639e5157f0" (UID: "2604e553-ae64-4fdf-ba10-1a639e5157f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.681067 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.680979 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-data" pod "2604e553-ae64-4fdf-ba10-1a639e5157f0" (UID: "2604e553-ae64-4fdf-ba10-1a639e5157f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.681184 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681164 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-credential-socket\") pod \"2604e553-ae64-4fdf-ba10-1a639e5157f0\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " Oct 31 13:16:14.681249 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681208 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-data\") pod \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " Oct 31 13:16:14.681249 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681238 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-socket\") pod \"2604e553-ae64-4fdf-ba10-1a639e5157f0\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " Oct 31 13:16:14.681351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681279 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-token\" (UniqueName: \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-token\") pod \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " Oct 31 13:16:14.681351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681298 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-envoy\") pod \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " Oct 31 13:16:14.681351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681325 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2604e553-ae64-4fdf-ba10-1a639e5157f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2604e553-ae64-4fdf-ba10-1a639e5157f0-istiod-ca-cert\") pod \"2604e553-ae64-4fdf-ba10-1a639e5157f0\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " Oct 31 13:16:14.681516 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681363 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-socket\") pod \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " Oct 31 13:16:14.681516 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681412 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-credential-socket" pod "2604e553-ae64-4fdf-ba10-1a639e5157f0" (UID: "2604e553-ae64-4fdf-ba10-1a639e5157f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.681516 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681450 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-data" pod "c48e0675-ee4f-4089-adcd-8c6eff556d2d" (UID: "c48e0675-ee4f-4089-adcd-8c6eff556d2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.681687 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681594 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-kube-api-access-c55sn\" (UniqueName: \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-kube-api-access-c55sn\") pod \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " Oct 31 13:16:14.681687 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681649 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-credential-socket\") pod \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " Oct 31 13:16:14.681687 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681598 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-socket" pod "2604e553-ae64-4fdf-ba10-1a639e5157f0" (UID: "2604e553-ae64-4fdf-ba10-1a639e5157f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.681840 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681701 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-certs\") pod \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " Oct 31 13:16:14.681840 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681730 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-podinfo\") pod \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " Oct 31 13:16:14.681840 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681799 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2604e553-ae64-4fdf-ba10-1a639e5157f0-istiod-ca-cert" pod "2604e553-ae64-4fdf-ba10-1a639e5157f0" (UID: "2604e553-ae64-4fdf-ba10-1a639e5157f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:14.681840 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681812 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-kube-api-access-s7c8d\" (UniqueName: \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-kube-api-access-s7c8d\") pod \"2604e553-ae64-4fdf-ba10-1a639e5157f0\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " Oct 31 13:16:14.682038 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681866 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-token\" (UniqueName: \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-token\") pod \"2604e553-ae64-4fdf-ba10-1a639e5157f0\" (UID: \"2604e553-ae64-4fdf-ba10-1a639e5157f0\") " Oct 31 13:16:14.682038 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.681952 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istiod-ca-cert\") pod \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\" (UID: \"c48e0675-ee4f-4089-adcd-8c6eff556d2d\") " Oct 31 13:16:14.682211 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682186 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-socket" pod "c48e0675-ee4f-4089-adcd-8c6eff556d2d" (UID: "c48e0675-ee4f-4089-adcd-8c6eff556d2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.682567 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682515 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-credential-socket" pod "c48e0675-ee4f-4089-adcd-8c6eff556d2d" (UID: "c48e0675-ee4f-4089-adcd-8c6eff556d2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.682672 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682645 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istiod-ca-cert" pod "c48e0675-ee4f-4089-adcd-8c6eff556d2d" (UID: "c48e0675-ee4f-4089-adcd-8c6eff556d2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:14.682672 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682659 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.682783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682683 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.682783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682691 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-certs" pod "c48e0675-ee4f-4089-adcd-8c6eff556d2d" (UID: "c48e0675-ee4f-4089-adcd-8c6eff556d2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.682783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682699 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w5jl\" (UniqueName: \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-kube-api-access-2w5jl\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.682783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682717 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.682783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682733 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.682783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682748 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.682783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682762 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.682783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682775 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682789 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7eb1ab-7542-44aa-a93d-d3186844631a-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682806 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7eb1ab-7542-44aa-a93d-d3186844631a-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682820 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682834 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682849 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.682863 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d057adb-f72c-4327-b979-aa61fdf98068-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.683091 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7eb1ab-7542-44aa-a93d-d3186844631a-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.683114 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2604e553-ae64-4fdf-ba10-1a639e5157f0-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.683132 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.683146 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d057adb-f72c-4327-b979-aa61fdf98068-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.683159 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.683172 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.683188 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr6nf\" (UniqueName: \"kubernetes.io/projected/2d057adb-f72c-4327-b979-aa61fdf98068-kube-api-access-pr6nf\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.683991 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.683958 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-envoy" pod "2604e553-ae64-4fdf-ba10-1a639e5157f0" (UID: "2604e553-ae64-4fdf-ba10-1a639e5157f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.683991 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.683986 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-podinfo" pod "2604e553-ae64-4fdf-ba10-1a639e5157f0" (UID: "2604e553-ae64-4fdf-ba10-1a639e5157f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:14.684353 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.684312 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-envoy" pod "c48e0675-ee4f-4089-adcd-8c6eff556d2d" (UID: "c48e0675-ee4f-4089-adcd-8c6eff556d2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:14.684818 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.684789 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-kube-api-access-s7c8d" pod "2604e553-ae64-4fdf-ba10-1a639e5157f0" (UID: "2604e553-ae64-4fdf-ba10-1a639e5157f0"). InnerVolumeSpecName "kube-api-access-s7c8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:14.685163 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.685138 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-token" pod "c48e0675-ee4f-4089-adcd-8c6eff556d2d" (UID: "c48e0675-ee4f-4089-adcd-8c6eff556d2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:14.685238 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.685159 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-kube-api-access-c55sn" pod "c48e0675-ee4f-4089-adcd-8c6eff556d2d" (UID: "c48e0675-ee4f-4089-adcd-8c6eff556d2d"). InnerVolumeSpecName "kube-api-access-c55sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:14.685810 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.685789 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-token" pod "2604e553-ae64-4fdf-ba10-1a639e5157f0" (UID: "2604e553-ae64-4fdf-ba10-1a639e5157f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:14.686017 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.686000 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-podinfo" pod "c48e0675-ee4f-4089-adcd-8c6eff556d2d" (UID: "c48e0675-ee4f-4089-adcd-8c6eff556d2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:14.784383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.784162 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.784383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.784200 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.784383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.784216 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c55sn\" (UniqueName: \"kubernetes.io/projected/c48e0675-ee4f-4089-adcd-8c6eff556d2d-kube-api-access-c55sn\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.784383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.784230 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c48e0675-ee4f-4089-adcd-8c6eff556d2d-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.784383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.784244 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.784383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.784257 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7c8d\" (UniqueName: \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-kube-api-access-s7c8d\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.784383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.784271 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.784383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.784284 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c48e0675-ee4f-4089-adcd-8c6eff556d2d-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.784383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.784297 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:14.784383 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:14.784312 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2604e553-ae64-4fdf-ba10-1a639e5157f0-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:15.141384 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.141352 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l"] Oct 31 13:16:15.145445 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.145419 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7bf66c-hs59l"] Oct 31 13:16:15.167232 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.167192 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:15.167802 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.167257 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:15.177403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.177370 4542 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:15.177589 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.177427 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:15.190886 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.190858 4542 generic.go:356] "Generic (PLEG): container finished" podID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerID="b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86" exitCode=0 Oct 31 13:16:15.191001 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.190885 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" event={"ID":"2604e553-ae64-4fdf-ba10-1a639e5157f0","Type":"ContainerDied","Data":"b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86"} Oct 31 13:16:15.191001 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.190917 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" Oct 31 13:16:15.191001 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.190943 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs" event={"ID":"2604e553-ae64-4fdf-ba10-1a639e5157f0","Type":"ContainerDied","Data":"d9b45ca21d91c64d31c4c5b00939312ea4c5f31f5c96c34d0a5fefd62bec47b1"} Oct 31 13:16:15.191001 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.190973 4542 scope.go:122] "RemoveContainer" containerID="b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86" Oct 31 13:16:15.192461 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.192433 4542 generic.go:356] "Generic (PLEG): container finished" podID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerID="d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21" exitCode=0 Oct 31 13:16:15.192580 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.192460 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" event={"ID":"c48e0675-ee4f-4089-adcd-8c6eff556d2d","Type":"ContainerDied","Data":"d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21"} Oct 31 13:16:15.192580 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.192501 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" Oct 31 13:16:15.192580 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.192526 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs" event={"ID":"c48e0675-ee4f-4089-adcd-8c6eff556d2d","Type":"ContainerDied","Data":"6e4bc3f9469ab3ebae68c2283ea9a5fd3035339e009912fb701ab0067e52f27e"} Oct 31 13:16:15.194435 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.194412 4542 generic.go:356] "Generic (PLEG): container finished" podID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerID="283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4" exitCode=0 Oct 31 13:16:15.194582 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.194500 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" event={"ID":"5c7eb1ab-7542-44aa-a93d-d3186844631a","Type":"ContainerDied","Data":"283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4"} Oct 31 13:16:15.194582 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.194515 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" Oct 31 13:16:15.194582 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.194546 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf" event={"ID":"5c7eb1ab-7542-44aa-a93d-d3186844631a","Type":"ContainerDied","Data":"fde6b3a01fd91b868b556e6dcc733fd6abc0755e042a3c144a71be37dd37e6e2"} Oct 31 13:16:15.198150 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.198129 4542 generic.go:356] "Generic (PLEG): container finished" podID="2d057adb-f72c-4327-b979-aa61fdf98068" containerID="b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645" exitCode=0 Oct 31 13:16:15.198237 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.198180 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" Oct 31 13:16:15.198237 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.198215 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" event={"ID":"2d057adb-f72c-4327-b979-aa61fdf98068","Type":"ContainerDied","Data":"b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645"} Oct 31 13:16:15.198343 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.198263 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht" event={"ID":"2d057adb-f72c-4327-b979-aa61fdf98068","Type":"ContainerDied","Data":"6037b0b2c959e5218a3d88241997d4869e4bb93df8b68c62d98ee755d8d2476e"} Oct 31 13:16:15.203758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.203737 4542 scope.go:122] "RemoveContainer" containerID="b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86" Oct 31 13:16:15.204186 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:15.204059 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86\": container with ID starting with b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86 not found: ID does not exist" containerID="b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86" Oct 31 13:16:15.204186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.204090 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86"} err="failed to get container status \"b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86\": rpc error: code = NotFound desc = could not find container \"b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86\": container with ID starting with b017582ef28f4b6bb7d61951ba123a180e7eebdc889fc9b4c29ced5e689ccb86 not found: ID does not exist" Oct 31 13:16:15.204186 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.204113 4542 scope.go:122] "RemoveContainer" containerID="d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21" Oct 31 13:16:15.213921 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.213905 4542 scope.go:122] "RemoveContainer" containerID="d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21" Oct 31 13:16:15.214172 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:15.214152 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21\": container with ID starting with d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21 not found: ID does not exist" containerID="d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21" Oct 31 13:16:15.214223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.214179 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21"} err="failed to get container status \"d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21\": rpc error: code = NotFound desc = could not find container \"d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21\": container with ID starting with d2ca6cf79df59ea4f5fe213f2923dc4d97ea8f90602f656c1b5273b297208d21 not found: ID does not exist" Oct 31 13:16:15.214223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.214194 4542 scope.go:122] "RemoveContainer" containerID="283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4" Oct 31 13:16:15.222506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.222466 4542 scope.go:122] "RemoveContainer" containerID="f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7" Oct 31 13:16:15.230410 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.230388 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs"] Oct 31 13:16:15.230923 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.230899 4542 scope.go:122] "RemoveContainer" containerID="283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4" Oct 31 13:16:15.231183 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:15.231166 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4\": container with ID starting with 283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4 not found: ID does not exist" containerID="283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4" Oct 31 13:16:15.231260 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.231187 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4"} err="failed to get container status \"283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4\": rpc error: code = NotFound desc = could not find container \"283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4\": container with ID starting with 283de223a2e6c821400cc3b2b165429e68f710c737843cbc9e3d8243035769a4 not found: ID does not exist" Oct 31 13:16:15.231260 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.231203 4542 scope.go:122] "RemoveContainer" containerID="f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7" Oct 31 13:16:15.231526 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:15.231509 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7\": container with ID starting with f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7 not found: ID does not exist" containerID="f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7" Oct 31 13:16:15.231607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.231528 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7"} err="failed to get container status \"f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7\": rpc error: code = NotFound desc = could not find container \"f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7\": container with ID starting with f3dab702f28ef59ae95badb8d4b321d897a18bd427efd746f30e6036fb875eb7 not found: ID does not exist" Oct 31 13:16:15.231607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.231542 4542 scope.go:122] "RemoveContainer" containerID="b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645" Oct 31 13:16:15.238627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.238606 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-vj5rs"] Oct 31 13:16:15.253775 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.249088 4542 scope.go:122] "RemoveContainer" containerID="b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645" Oct 31 13:16:15.253775 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:15.249368 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645\": container with ID starting with b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645 not found: ID does not exist" containerID="b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645" Oct 31 13:16:15.253775 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.249394 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645"} err="failed to get container status \"b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645\": rpc error: code = NotFound desc = could not find container \"b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645\": container with ID starting with b6cd659e28b1fb935590443d5fd151210d9e9a225617ae6f3af4ca09561ff645 not found: ID does not exist" Oct 31 13:16:15.262715 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.262677 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf"] Oct 31 13:16:15.266614 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.266594 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-l42nf"] Oct 31 13:16:15.282190 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.282170 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht"] Oct 31 13:16:15.291143 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.291113 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" path="/var/lib/kubelet/pods/2f8e330f-01c1-435e-b684-56a1ed9208b7/volumes" Oct 31 13:16:15.291617 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.291598 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" path="/var/lib/kubelet/pods/32ddbe95-4c3e-494d-9257-accc94089b63/volumes" Oct 31 13:16:15.292043 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.292025 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bcbb4d-b174-4a1d-b5ce-6e28726bcf74" path="/var/lib/kubelet/pods/46bcbb4d-b174-4a1d-b5ce-6e28726bcf74/volumes" Oct 31 13:16:15.292801 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.292782 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" path="/var/lib/kubelet/pods/5c7eb1ab-7542-44aa-a93d-d3186844631a/volumes" Oct 31 13:16:15.293275 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.293257 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" path="/var/lib/kubelet/pods/c48e0675-ee4f-4089-adcd-8c6eff556d2d/volumes" Oct 31 13:16:15.293720 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.293702 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" path="/var/lib/kubelet/pods/eabf6be0-70ac-4a23-bce4-23a0047c0cfc/volumes" Oct 31 13:16:15.294412 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.294393 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" path="/var/lib/kubelet/pods/f067868a-a184-4ef4-9f5c-7de384be46ae/volumes" Oct 31 13:16:15.294725 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.294709 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-fkpht"] Oct 31 13:16:15.304693 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.304675 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs"] Oct 31 13:16:15.310684 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.310664 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-qx2fs"] Oct 31 13:16:15.914114 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.914079 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:15.914271 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:15.914137 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:16.417444 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:16.417406 4542 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:16.417828 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:16.417464 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:17.228687 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.224319 4542 generic.go:356] "Generic (PLEG): container finished" podID="f48b8206-4423-4c72-8ade-a37075d55f97" containerID="103b758424a7ec7fd142f8bff3a018b48d84fdef3ff7a5af3c11459abd0e860a" exitCode=0 Oct 31 13:16:17.228687 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.224412 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" event={"ID":"f48b8206-4423-4c72-8ade-a37075d55f97","Type":"ContainerDied","Data":"103b758424a7ec7fd142f8bff3a018b48d84fdef3ff7a5af3c11459abd0e860a"} Oct 31 13:16:17.228687 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.226697 4542 generic.go:356] "Generic (PLEG): container finished" podID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerID="b1127fccc9168a3056055ebb0fee7b34e48ff64e03e23eba1cea601d572fdab6" exitCode=0 Oct 31 13:16:17.228687 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.226812 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" event={"ID":"3d85d750-b72f-4664-b3b2-1a0f802a6d81","Type":"ContainerDied","Data":"b1127fccc9168a3056055ebb0fee7b34e48ff64e03e23eba1cea601d572fdab6"} Oct 31 13:16:17.232517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.229179 4542 generic.go:356] "Generic (PLEG): container finished" podID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerID="97ede513ae6cfd2cae07d632c12691bd4b52079aae5e43ca4ac55257408be486" exitCode=0 Oct 31 13:16:17.232517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.229279 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" event={"ID":"95b70fcb-394c-4c0d-baa8-d0c8323c0723","Type":"ContainerDied","Data":"97ede513ae6cfd2cae07d632c12691bd4b52079aae5e43ca4ac55257408be486"} Oct 31 13:16:17.232517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.231403 4542 generic.go:356] "Generic (PLEG): container finished" podID="e02d8eed-d690-4d65-981a-03b5a4331852" containerID="0af2b04ccc271ab6f25bf533782c5535b14cb83381f42aa675437ce156786c87" exitCode=0 Oct 31 13:16:17.232517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.231498 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" event={"ID":"e02d8eed-d690-4d65-981a-03b5a4331852","Type":"ContainerDied","Data":"0af2b04ccc271ab6f25bf533782c5535b14cb83381f42aa675437ce156786c87"} Oct 31 13:16:17.307633 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.306777 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" path="/var/lib/kubelet/pods/2604e553-ae64-4fdf-ba10-1a639e5157f0/volumes" Oct 31 13:16:17.307633 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.307314 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" path="/var/lib/kubelet/pods/2d057adb-f72c-4327-b979-aa61fdf98068/volumes" Oct 31 13:16:17.415212 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.413636 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:16:17.515144 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.515033 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-kube-api-access-sc7ns\" (UniqueName: \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-kube-api-access-sc7ns\") pod \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " Oct 31 13:16:17.515725 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.515689 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-token\" (UniqueName: \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-token\") pod \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " Oct 31 13:16:17.515955 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.515919 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istiod-ca-cert\") pod \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " Oct 31 13:16:17.516109 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.516065 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-podinfo\") pod \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " Oct 31 13:16:17.516262 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.516195 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-envoy\") pod \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " Oct 31 13:16:17.516491 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.516397 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-data\") pod \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " Oct 31 13:16:17.516722 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.516629 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-credential-socket\") pod \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " Oct 31 13:16:17.516876 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.516802 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-certs\") pod \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " Oct 31 13:16:17.516876 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.516847 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-socket\") pod \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\" (UID: \"3d85d750-b72f-4664-b3b2-1a0f802a6d81\") " Oct 31 13:16:17.523751 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.523687 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-credential-socket" pod "3d85d750-b72f-4664-b3b2-1a0f802a6d81" (UID: "3d85d750-b72f-4664-b3b2-1a0f802a6d81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.524205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.523857 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-data" pod "3d85d750-b72f-4664-b3b2-1a0f802a6d81" (UID: "3d85d750-b72f-4664-b3b2-1a0f802a6d81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.524205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.523933 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-socket" pod "3d85d750-b72f-4664-b3b2-1a0f802a6d81" (UID: "3d85d750-b72f-4664-b3b2-1a0f802a6d81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.524205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.524067 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istiod-ca-cert" pod "3d85d750-b72f-4664-b3b2-1a0f802a6d81" (UID: "3d85d750-b72f-4664-b3b2-1a0f802a6d81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:17.524205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.524185 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-certs" pod "3d85d750-b72f-4664-b3b2-1a0f802a6d81" (UID: "3d85d750-b72f-4664-b3b2-1a0f802a6d81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.524957 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.524920 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-podinfo" pod "3d85d750-b72f-4664-b3b2-1a0f802a6d81" (UID: "3d85d750-b72f-4664-b3b2-1a0f802a6d81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:17.527805 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.527769 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-envoy" pod "3d85d750-b72f-4664-b3b2-1a0f802a6d81" (UID: "3d85d750-b72f-4664-b3b2-1a0f802a6d81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.528293 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.528256 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-token" pod "3d85d750-b72f-4664-b3b2-1a0f802a6d81" (UID: "3d85d750-b72f-4664-b3b2-1a0f802a6d81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:17.528976 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.528941 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-kube-api-access-sc7ns" pod "3d85d750-b72f-4664-b3b2-1a0f802a6d81" (UID: "3d85d750-b72f-4664-b3b2-1a0f802a6d81"). InnerVolumeSpecName "kube-api-access-sc7ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:17.556176 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.553651 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:16:17.561415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.560879 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:16:17.571257 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.571186 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:16:17.618264 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618228 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-data\") pod \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " Oct 31 13:16:17.618453 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618287 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-socket\") pod \"f48b8206-4423-4c72-8ade-a37075d55f97\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " Oct 31 13:16:17.618453 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618320 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-socket\") pod \"e02d8eed-d690-4d65-981a-03b5a4331852\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " Oct 31 13:16:17.618453 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618371 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-kube-api-access-b4fjl\" (UniqueName: \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-kube-api-access-b4fjl\") pod \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " Oct 31 13:16:17.618453 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618406 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f48b8206-4423-4c72-8ade-a37075d55f97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48b8206-4423-4c72-8ade-a37075d55f97-istiod-ca-cert\") pod \"f48b8206-4423-4c72-8ade-a37075d55f97\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " Oct 31 13:16:17.618453 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618446 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e02d8eed-d690-4d65-981a-03b5a4331852-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02d8eed-d690-4d65-981a-03b5a4331852-istiod-ca-cert\") pod \"e02d8eed-d690-4d65-981a-03b5a4331852\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " Oct 31 13:16:17.618779 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618524 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-socket\") pod \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " Oct 31 13:16:17.618779 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618572 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-certs\") pod \"f48b8206-4423-4c72-8ade-a37075d55f97\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " Oct 31 13:16:17.618779 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618611 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-kube-api-access-vwlxt\" (UniqueName: \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-kube-api-access-vwlxt\") pod \"e02d8eed-d690-4d65-981a-03b5a4331852\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " Oct 31 13:16:17.618779 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618644 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-podinfo\") pod \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " Oct 31 13:16:17.618779 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618672 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-istio-token\" (UniqueName: \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-istio-token\") pod \"e02d8eed-d690-4d65-981a-03b5a4331852\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " Oct 31 13:16:17.618779 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618711 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-envoy\") pod \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " Oct 31 13:16:17.618779 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618744 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-envoy\") pod \"e02d8eed-d690-4d65-981a-03b5a4331852\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " Oct 31 13:16:17.618779 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618776 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f48b8206-4423-4c72-8ade-a37075d55f97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48b8206-4423-4c72-8ade-a37075d55f97-istio-podinfo\") pod \"f48b8206-4423-4c72-8ade-a37075d55f97\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618828 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-credential-socket\") pod \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618863 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e02d8eed-d690-4d65-981a-03b5a4331852-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02d8eed-d690-4d65-981a-03b5a4331852-istio-podinfo\") pod \"e02d8eed-d690-4d65-981a-03b5a4331852\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618909 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-envoy\") pod \"f48b8206-4423-4c72-8ade-a37075d55f97\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618954 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-certs\") pod \"e02d8eed-d690-4d65-981a-03b5a4331852\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618975 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-certs" pod "f48b8206-4423-4c72-8ade-a37075d55f97" (UID: "f48b8206-4423-4c72-8ade-a37075d55f97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.618993 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-credential-socket\") pod \"f48b8206-4423-4c72-8ade-a37075d55f97\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.619025 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-certs\") pod \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.619061 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-data\") pod \"f48b8206-4423-4c72-8ade-a37075d55f97\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.619095 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-kube-api-access-zrr8x\" (UniqueName: \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-kube-api-access-zrr8x\") pod \"f48b8206-4423-4c72-8ade-a37075d55f97\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.619185 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-socket" pod "e02d8eed-d690-4d65-981a-03b5a4331852" (UID: "e02d8eed-d690-4d65-981a-03b5a4331852"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.619227 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.619206 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-istio-token\" (UniqueName: \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-istio-token\") pod \"f48b8206-4423-4c72-8ade-a37075d55f97\" (UID: \"f48b8206-4423-4c72-8ade-a37075d55f97\") " Oct 31 13:16:17.619919 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.619561 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-data" pod "95b70fcb-394c-4c0d-baa8-d0c8323c0723" (UID: "95b70fcb-394c-4c0d-baa8-d0c8323c0723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.619919 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.619839 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istiod-ca-cert\") pod \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " Oct 31 13:16:17.620044 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.619916 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-credential-socket\") pod \"e02d8eed-d690-4d65-981a-03b5a4331852\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " Oct 31 13:16:17.620044 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.619923 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02d8eed-d690-4d65-981a-03b5a4331852-istiod-ca-cert" pod "e02d8eed-d690-4d65-981a-03b5a4331852" (UID: "e02d8eed-d690-4d65-981a-03b5a4331852"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:17.620044 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.619974 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-token\" (UniqueName: \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-token\") pod \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\" (UID: \"95b70fcb-394c-4c0d-baa8-d0c8323c0723\") " Oct 31 13:16:17.620213 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620048 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-data\") pod \"e02d8eed-d690-4d65-981a-03b5a4331852\" (UID: \"e02d8eed-d690-4d65-981a-03b5a4331852\") " Oct 31 13:16:17.620277 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620220 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48b8206-4423-4c72-8ade-a37075d55f97-istiod-ca-cert" pod "f48b8206-4423-4c72-8ade-a37075d55f97" (UID: "f48b8206-4423-4c72-8ade-a37075d55f97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620421 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-credential-socket" pod "95b70fcb-394c-4c0d-baa8-d0c8323c0723" (UID: "95b70fcb-394c-4c0d-baa8-d0c8323c0723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620606 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-data" pod "e02d8eed-d690-4d65-981a-03b5a4331852" (UID: "e02d8eed-d690-4d65-981a-03b5a4331852"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620691 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620712 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620726 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620740 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620753 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-socket" pod "95b70fcb-394c-4c0d-baa8-d0c8323c0723" (UID: "95b70fcb-394c-4c0d-baa8-d0c8323c0723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620755 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620781 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620799 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d85d750-b72f-4664-b3b2-1a0f802a6d81-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620815 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc7ns\" (UniqueName: \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-kube-api-access-sc7ns\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620831 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620847 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620861 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620876 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48b8206-4423-4c72-8ade-a37075d55f97-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620891 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02d8eed-d690-4d65-981a-03b5a4331852-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620905 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d85d750-b72f-4664-b3b2-1a0f802a6d81-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.620921 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.621007 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-socket" pod "f48b8206-4423-4c72-8ade-a37075d55f97" (UID: "f48b8206-4423-4c72-8ade-a37075d55f97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.621220 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-credential-socket" pod "f48b8206-4423-4c72-8ade-a37075d55f97" (UID: "f48b8206-4423-4c72-8ade-a37075d55f97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.621252 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istiod-ca-cert" pod "95b70fcb-394c-4c0d-baa8-d0c8323c0723" (UID: "95b70fcb-394c-4c0d-baa8-d0c8323c0723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:17.621518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.621442 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-credential-socket" pod "e02d8eed-d690-4d65-981a-03b5a4331852" (UID: "e02d8eed-d690-4d65-981a-03b5a4331852"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.622673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.622198 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-certs" pod "95b70fcb-394c-4c0d-baa8-d0c8323c0723" (UID: "95b70fcb-394c-4c0d-baa8-d0c8323c0723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.622673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.622228 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-certs" pod "e02d8eed-d690-4d65-981a-03b5a4331852" (UID: "e02d8eed-d690-4d65-981a-03b5a4331852"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.622673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.622384 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-data" pod "f48b8206-4423-4c72-8ade-a37075d55f97" (UID: "f48b8206-4423-4c72-8ade-a37075d55f97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.627360 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.627279 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-kube-api-access-b4fjl" pod "95b70fcb-394c-4c0d-baa8-d0c8323c0723" (UID: "95b70fcb-394c-4c0d-baa8-d0c8323c0723"). InnerVolumeSpecName "kube-api-access-b4fjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:17.630530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.630505 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-podinfo" pod "95b70fcb-394c-4c0d-baa8-d0c8323c0723" (UID: "95b70fcb-394c-4c0d-baa8-d0c8323c0723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:17.635561 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.634776 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f48b8206-4423-4c72-8ade-a37075d55f97-istio-podinfo" pod "f48b8206-4423-4c72-8ade-a37075d55f97" (UID: "f48b8206-4423-4c72-8ade-a37075d55f97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:17.642537 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.642500 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-istio-token" pod "e02d8eed-d690-4d65-981a-03b5a4331852" (UID: "e02d8eed-d690-4d65-981a-03b5a4331852"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:17.642692 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.642660 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-envoy" pod "e02d8eed-d690-4d65-981a-03b5a4331852" (UID: "e02d8eed-d690-4d65-981a-03b5a4331852"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.643519 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.643462 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-envoy" pod "f48b8206-4423-4c72-8ade-a37075d55f97" (UID: "f48b8206-4423-4c72-8ade-a37075d55f97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.643629 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.643607 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-token" pod "95b70fcb-394c-4c0d-baa8-d0c8323c0723" (UID: "95b70fcb-394c-4c0d-baa8-d0c8323c0723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:17.643761 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.643739 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e02d8eed-d690-4d65-981a-03b5a4331852-istio-podinfo" pod "e02d8eed-d690-4d65-981a-03b5a4331852" (UID: "e02d8eed-d690-4d65-981a-03b5a4331852"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:17.643952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.643923 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-kube-api-access-vwlxt" pod "e02d8eed-d690-4d65-981a-03b5a4331852" (UID: "e02d8eed-d690-4d65-981a-03b5a4331852"). InnerVolumeSpecName "kube-api-access-vwlxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:17.643952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.643930 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-istio-token" pod "f48b8206-4423-4c72-8ade-a37075d55f97" (UID: "f48b8206-4423-4c72-8ade-a37075d55f97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:17.645402 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.645375 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-kube-api-access-zrr8x" pod "f48b8206-4423-4c72-8ade-a37075d55f97" (UID: "f48b8206-4423-4c72-8ade-a37075d55f97"). InnerVolumeSpecName "kube-api-access-zrr8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:17.645845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.645813 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-envoy" pod "95b70fcb-394c-4c0d-baa8-d0c8323c0723" (UID: "95b70fcb-394c-4c0d-baa8-d0c8323c0723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722538 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722584 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722601 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4fjl\" (UniqueName: \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-kube-api-access-b4fjl\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722617 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722635 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwlxt\" (UniqueName: \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-kube-api-access-vwlxt\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722649 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722663 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02d8eed-d690-4d65-981a-03b5a4331852-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722677 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722692 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722707 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48b8206-4423-4c72-8ade-a37075d55f97-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722721 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02d8eed-d690-4d65-981a-03b5a4331852-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722733 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722747 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722761 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722775 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b70fcb-394c-4c0d-baa8-d0c8323c0723-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722789 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48b8206-4423-4c72-8ade-a37075d55f97-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722805 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrr8x\" (UniqueName: \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-kube-api-access-zrr8x\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722820 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48b8206-4423-4c72-8ade-a37075d55f97-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722834 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722848 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02d8eed-d690-4d65-981a-03b5a4331852-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:17.722889 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:17.722862 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b70fcb-394c-4c0d-baa8-d0c8323c0723-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.332342 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.332168 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" event={"ID":"f48b8206-4423-4c72-8ade-a37075d55f97","Type":"ContainerDied","Data":"a1dae318c2981c386923417a0da6611f491e5ecedb966454563be33b8d270c29"} Oct 31 13:16:18.332342 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.332244 4542 scope.go:122] "RemoveContainer" containerID="103b758424a7ec7fd142f8bff3a018b48d84fdef3ff7a5af3c11459abd0e860a" Oct 31 13:16:18.332524 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.332348 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5" Oct 31 13:16:18.377581 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.358755 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" event={"ID":"3d85d750-b72f-4664-b3b2-1a0f802a6d81","Type":"ContainerDied","Data":"b6906c5c53c4f60876595610667f275091c3cd1af67ef33acaf2e59d02a89878"} Oct 31 13:16:18.377581 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.358898 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb" Oct 31 13:16:18.397571 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.394012 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" event={"ID":"95b70fcb-394c-4c0d-baa8-d0c8323c0723","Type":"ContainerDied","Data":"300c4d51fcedd828742c7bc5711acf40606a3b0b5f78de502c35dbd769806d3a"} Oct 31 13:16:18.397571 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.394144 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6" Oct 31 13:16:18.408141 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.407193 4542 generic.go:356] "Generic (PLEG): container finished" podID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerID="b48ebd6102c9beb09791a08c40b5fd062244248748fcf5d514de4ea0671637ed" exitCode=0 Oct 31 13:16:18.408141 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.407294 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" event={"ID":"d6150f29-b21c-4142-8b99-7fb0af958d0d","Type":"ContainerDied","Data":"b48ebd6102c9beb09791a08c40b5fd062244248748fcf5d514de4ea0671637ed"} Oct 31 13:16:18.410550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.409222 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" event={"ID":"e02d8eed-d690-4d65-981a-03b5a4331852","Type":"ContainerDied","Data":"c878a82638ce23653ed7f997cd46b50a04e781717ea53ac82b2238b487ddbddc"} Oct 31 13:16:18.410550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.409231 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c" Oct 31 13:16:18.414966 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.414938 4542 scope.go:122] "RemoveContainer" containerID="b1127fccc9168a3056055ebb0fee7b34e48ff64e03e23eba1cea601d572fdab6" Oct 31 13:16:18.425164 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.425082 4542 generic.go:356] "Generic (PLEG): container finished" podID="b015433d-b413-49a1-a515-f903370ba0e0" containerID="7e6c4dab7f1c7226a882c66666f84a522da59b8ff7c77fc92cc4386b627b7bf4" exitCode=0 Oct 31 13:16:18.425273 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.425190 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" event={"ID":"b015433d-b413-49a1-a515-f903370ba0e0","Type":"ContainerDied","Data":"7e6c4dab7f1c7226a882c66666f84a522da59b8ff7c77fc92cc4386b627b7bf4"} Oct 31 13:16:18.432327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.430916 4542 generic.go:356] "Generic (PLEG): container finished" podID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerID="2b531afbb82da126495810f2dc90b77be0c542ea789237ad4b4b11d3943eda13" exitCode=0 Oct 31 13:16:18.432327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.431150 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" event={"ID":"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40","Type":"ContainerDied","Data":"2b531afbb82da126495810f2dc90b77be0c542ea789237ad4b4b11d3943eda13"} Oct 31 13:16:18.445842 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.438075 4542 generic.go:356] "Generic (PLEG): container finished" podID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerID="39a0dcaf7e4ce535078f9c07bb52ceb47cc891394c50ece4404e70bcceb98b2b" exitCode=0 Oct 31 13:16:18.445842 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.438132 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" event={"ID":"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f","Type":"ContainerDied","Data":"39a0dcaf7e4ce535078f9c07bb52ceb47cc891394c50ece4404e70bcceb98b2b"} Oct 31 13:16:18.448381 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.448322 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5"] Oct 31 13:16:18.450885 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.450845 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-7nzk5"] Oct 31 13:16:18.453977 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.453956 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb"] Oct 31 13:16:18.465708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.465686 4542 scope.go:122] "RemoveContainer" containerID="97ede513ae6cfd2cae07d632c12691bd4b52079aae5e43ca4ac55257408be486" Oct 31 13:16:18.466928 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.466903 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-m6flb"] Oct 31 13:16:18.485547 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.485523 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6"] Oct 31 13:16:18.487632 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.487609 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-45hc6"] Oct 31 13:16:18.487787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.487771 4542 scope.go:122] "RemoveContainer" containerID="0af2b04ccc271ab6f25bf533782c5535b14cb83381f42aa675437ce156786c87" Oct 31 13:16:18.506575 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.506550 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c"] Oct 31 13:16:18.509050 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.508955 4542 scope.go:122] "RemoveContainer" containerID="d10957a4dc081b7d897ab00aecb54c4036149d642ccc8007213b0458f725f1fc" Oct 31 13:16:18.509432 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.509411 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-fp29c"] Oct 31 13:16:18.552676 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.552653 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:16:18.612376 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.612336 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:16:18.639236 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.638825 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-kube-api-access-n962m\" (UniqueName: \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-kube-api-access-n962m\") pod \"b015433d-b413-49a1-a515-f903370ba0e0\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " Oct 31 13:16:18.639236 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.638907 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-credential-socket\") pod \"d6150f29-b21c-4142-8b99-7fb0af958d0d\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " Oct 31 13:16:18.639236 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.638940 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-data\") pod \"d6150f29-b21c-4142-8b99-7fb0af958d0d\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " Oct 31 13:16:18.641530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.639609 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-kube-api-access-rrz94\" (UniqueName: \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-kube-api-access-rrz94\") pod \"d6150f29-b21c-4142-8b99-7fb0af958d0d\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " Oct 31 13:16:18.641530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.639666 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-data\") pod \"b015433d-b413-49a1-a515-f903370ba0e0\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " Oct 31 13:16:18.641530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.639716 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b015433d-b413-49a1-a515-f903370ba0e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b015433d-b413-49a1-a515-f903370ba0e0-istio-podinfo\") pod \"b015433d-b413-49a1-a515-f903370ba0e0\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " Oct 31 13:16:18.641530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.639771 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-envoy\") pod \"b015433d-b413-49a1-a515-f903370ba0e0\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " Oct 31 13:16:18.641530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.639805 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-podinfo\") pod \"d6150f29-b21c-4142-8b99-7fb0af958d0d\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " Oct 31 13:16:18.641530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.639873 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-data" pod "d6150f29-b21c-4142-8b99-7fb0af958d0d" (UID: "d6150f29-b21c-4142-8b99-7fb0af958d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.641530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.639912 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-credential-socket" pod "d6150f29-b21c-4142-8b99-7fb0af958d0d" (UID: "d6150f29-b21c-4142-8b99-7fb0af958d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.644517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.641971 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6150f29-b21c-4142-8b99-7fb0af958d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6150f29-b21c-4142-8b99-7fb0af958d0d-istiod-ca-cert\") pod \"d6150f29-b21c-4142-8b99-7fb0af958d0d\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " Oct 31 13:16:18.644517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.643059 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-data" pod "b015433d-b413-49a1-a515-f903370ba0e0" (UID: "b015433d-b413-49a1-a515-f903370ba0e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.644517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.643270 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-socket\") pod \"d6150f29-b21c-4142-8b99-7fb0af958d0d\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " Oct 31 13:16:18.644517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.643363 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-certs\") pod \"d6150f29-b21c-4142-8b99-7fb0af958d0d\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " Oct 31 13:16:18.644517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.643408 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-istio-token\" (UniqueName: \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-istio-token\") pod \"b015433d-b413-49a1-a515-f903370ba0e0\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " Oct 31 13:16:18.644517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.643437 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-envoy\") pod \"d6150f29-b21c-4142-8b99-7fb0af958d0d\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " Oct 31 13:16:18.644517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.643491 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b015433d-b413-49a1-a515-f903370ba0e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b015433d-b413-49a1-a515-f903370ba0e0-istiod-ca-cert\") pod \"b015433d-b413-49a1-a515-f903370ba0e0\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " Oct 31 13:16:18.644517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.643527 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-credential-socket\") pod \"b015433d-b413-49a1-a515-f903370ba0e0\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " Oct 31 13:16:18.644517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.643560 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-token\") pod \"d6150f29-b21c-4142-8b99-7fb0af958d0d\" (UID: \"d6150f29-b21c-4142-8b99-7fb0af958d0d\") " Oct 31 13:16:18.645083 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.645055 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-socket\") pod \"b015433d-b413-49a1-a515-f903370ba0e0\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " Oct 31 13:16:18.645208 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.645143 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-certs\") pod \"b015433d-b413-49a1-a515-f903370ba0e0\" (UID: \"b015433d-b413-49a1-a515-f903370ba0e0\") " Oct 31 13:16:18.647029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.646072 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-credential-socket" pod "b015433d-b413-49a1-a515-f903370ba0e0" (UID: "b015433d-b413-49a1-a515-f903370ba0e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.647029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.646256 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.647029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.646278 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.647029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.646295 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.647029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.646311 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.647029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.646677 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-socket" pod "d6150f29-b21c-4142-8b99-7fb0af958d0d" (UID: "d6150f29-b21c-4142-8b99-7fb0af958d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.647029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.646741 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-socket" pod "b015433d-b413-49a1-a515-f903370ba0e0" (UID: "b015433d-b413-49a1-a515-f903370ba0e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.647029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.646912 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-certs" pod "d6150f29-b21c-4142-8b99-7fb0af958d0d" (UID: "d6150f29-b21c-4142-8b99-7fb0af958d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.647029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.646972 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b015433d-b413-49a1-a515-f903370ba0e0-istiod-ca-cert" pod "b015433d-b413-49a1-a515-f903370ba0e0" (UID: "b015433d-b413-49a1-a515-f903370ba0e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:18.647029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.647027 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-certs" pod "b015433d-b413-49a1-a515-f903370ba0e0" (UID: "b015433d-b413-49a1-a515-f903370ba0e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.647646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.647596 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-podinfo" pod "d6150f29-b21c-4142-8b99-7fb0af958d0d" (UID: "d6150f29-b21c-4142-8b99-7fb0af958d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:18.647712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.647684 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-istio-token" pod "b015433d-b413-49a1-a515-f903370ba0e0" (UID: "b015433d-b413-49a1-a515-f903370ba0e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:18.647754 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.647696 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6150f29-b21c-4142-8b99-7fb0af958d0d-istiod-ca-cert" pod "d6150f29-b21c-4142-8b99-7fb0af958d0d" (UID: "d6150f29-b21c-4142-8b99-7fb0af958d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:18.649719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.649400 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-token" pod "d6150f29-b21c-4142-8b99-7fb0af958d0d" (UID: "d6150f29-b21c-4142-8b99-7fb0af958d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:18.649719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.649585 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b015433d-b413-49a1-a515-f903370ba0e0-istio-podinfo" pod "b015433d-b413-49a1-a515-f903370ba0e0" (UID: "b015433d-b413-49a1-a515-f903370ba0e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:18.649719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.649683 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-kube-api-access-n962m" pod "b015433d-b413-49a1-a515-f903370ba0e0" (UID: "b015433d-b413-49a1-a515-f903370ba0e0"). InnerVolumeSpecName "kube-api-access-n962m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:18.650094 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.650063 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-envoy" pod "d6150f29-b21c-4142-8b99-7fb0af958d0d" (UID: "d6150f29-b21c-4142-8b99-7fb0af958d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.651343 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.651319 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-kube-api-access-rrz94" pod "d6150f29-b21c-4142-8b99-7fb0af958d0d" (UID: "d6150f29-b21c-4142-8b99-7fb0af958d0d"). InnerVolumeSpecName "kube-api-access-rrz94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:18.651441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.651357 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-envoy" pod "b015433d-b413-49a1-a515-f903370ba0e0" (UID: "b015433d-b413-49a1-a515-f903370ba0e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.668770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.666850 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:16:18.672560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.669919 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747123 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-credential-socket\") pod \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747199 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-envoy\") pod \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747243 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-certs\") pod \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747292 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-podinfo\") pod \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747353 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-token\" (UniqueName: \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-token\") pod \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747381 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-credential-socket" pod "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" (UID: "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747389 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-kube-api-access-k579r\" (UniqueName: \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-kube-api-access-k579r\") pod \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747417 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-socket\") pod \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747447 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-token\" (UniqueName: \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-token\") pod \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747513 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-socket\") pod \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747544 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-credential-socket\") pod \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747576 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-data\") pod \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747636 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istiod-ca-cert\") pod \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747674 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-certs\") pod \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747736 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-envoy\") pod \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\" (UID: \"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747789 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-kube-api-access-qzjbm\" (UniqueName: \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-kube-api-access-qzjbm\") pod \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747823 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-podinfo\") pod \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747849 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-data\") pod \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.747878 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istiod-ca-cert\") pod \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\" (UID: \"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40\") " Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748163 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-credential-socket" pod "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" (UID: "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.748371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748390 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrz94\" (UniqueName: \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-kube-api-access-rrz94\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748406 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748416 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b015433d-b413-49a1-a515-f903370ba0e0-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748427 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748436 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748448 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6150f29-b21c-4142-8b99-7fb0af958d0d-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748461 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748489 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748504 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748515 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748524 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748532 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b015433d-b413-49a1-a515-f903370ba0e0-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748540 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6150f29-b21c-4142-8b99-7fb0af958d0d-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748548 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748556 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b015433d-b413-49a1-a515-f903370ba0e0-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748565 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n962m\" (UniqueName: \"kubernetes.io/projected/b015433d-b413-49a1-a515-f903370ba0e0-kube-api-access-n962m\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748701 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-socket" pod "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" (UID: "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.748940 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istiod-ca-cert" pod "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" (UID: "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:18.749563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.749094 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-data" pod "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" (UID: "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.750408 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.749691 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istiod-ca-cert" pod "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" (UID: "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:18.750408 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.749841 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-socket" pod "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" (UID: "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.750408 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.749986 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-data" pod "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" (UID: "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.750408 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.750111 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-certs" pod "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" (UID: "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.750408 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.750170 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-certs" pod "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" (UID: "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.751759 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.751732 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-kube-api-access-qzjbm" pod "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" (UID: "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40"). InnerVolumeSpecName "kube-api-access-qzjbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:18.752148 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.752112 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-token" pod "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" (UID: "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:18.754368 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.754330 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-token" pod "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" (UID: "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:18.755809 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.755784 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-envoy" pod "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" (UID: "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.756531 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.756496 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-podinfo" pod "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" (UID: "b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:18.757672 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.757634 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-podinfo" pod "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" (UID: "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:18.758172 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.758145 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-kube-api-access-k579r" pod "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" (UID: "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f"). InnerVolumeSpecName "kube-api-access-k579r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:18.758435 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.758411 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-envoy" pod "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" (UID: "cbe40c82-7a02-4fa3-8987-3f88fdf3f60f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:18.849182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849152 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849182 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzjbm\" (UniqueName: \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-kube-api-access-qzjbm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849199 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849208 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849216 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849224 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849232 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849240 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849252 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849263 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k579r\" (UniqueName: \"kubernetes.io/projected/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-kube-api-access-k579r\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849271 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849280 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849288 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849297 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849305 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:18.849386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:18.849313 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:19.292016 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.291979 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" path="/var/lib/kubelet/pods/3d85d750-b72f-4664-b3b2-1a0f802a6d81/volumes" Oct 31 13:16:19.292630 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.292607 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" path="/var/lib/kubelet/pods/95b70fcb-394c-4c0d-baa8-d0c8323c0723/volumes" Oct 31 13:16:19.293056 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.293042 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" path="/var/lib/kubelet/pods/e02d8eed-d690-4d65-981a-03b5a4331852/volumes" Oct 31 13:16:19.293914 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.293887 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" path="/var/lib/kubelet/pods/f48b8206-4423-4c72-8ade-a37075d55f97/volumes" Oct 31 13:16:19.447841 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.447760 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" event={"ID":"b015433d-b413-49a1-a515-f903370ba0e0","Type":"ContainerDied","Data":"90986d56ce35d781595089e921e0e96c27367cb86501d7c58c1e276b84e4a8c9"} Oct 31 13:16:19.447995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.447843 4542 scope.go:122] "RemoveContainer" containerID="7e6c4dab7f1c7226a882c66666f84a522da59b8ff7c77fc92cc4386b627b7bf4" Oct 31 13:16:19.447995 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.447776 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r" Oct 31 13:16:19.452254 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.452234 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" Oct 31 13:16:19.452506 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.452238 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln" event={"ID":"b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40","Type":"ContainerDied","Data":"7fa68ceaf2facb0b02ccf7682b988095272968b587b5930d213724aa2710c5cd"} Oct 31 13:16:19.454969 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.454943 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" Oct 31 13:16:19.455072 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.454962 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf" event={"ID":"cbe40c82-7a02-4fa3-8987-3f88fdf3f60f","Type":"ContainerDied","Data":"c9d1aba92ba1bf89412ad9a496b0c876254a0a385691075cd4eab4923f7bf887"} Oct 31 13:16:19.459920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.459891 4542 scope.go:122] "RemoveContainer" containerID="2b531afbb82da126495810f2dc90b77be0c542ea789237ad4b4b11d3943eda13" Oct 31 13:16:19.463122 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.463096 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" event={"ID":"d6150f29-b21c-4142-8b99-7fb0af958d0d","Type":"ContainerDied","Data":"ee72f05c9cae828b29886e57028aaf68e55a9272027f477122f9ff721466a34d"} Oct 31 13:16:19.463122 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.463078 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq" Oct 31 13:16:19.469821 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.469803 4542 scope.go:122] "RemoveContainer" containerID="39a0dcaf7e4ce535078f9c07bb52ceb47cc891394c50ece4404e70bcceb98b2b" Oct 31 13:16:19.477026 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.477001 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r"] Oct 31 13:16:19.480466 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.480446 4542 scope.go:122] "RemoveContainer" containerID="b48ebd6102c9beb09791a08c40b5fd062244248748fcf5d514de4ea0671637ed" Oct 31 13:16:19.481923 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.481631 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-vbd8r"] Oct 31 13:16:19.496724 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.496699 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf"] Oct 31 13:16:19.501054 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.501035 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-h56zf"] Oct 31 13:16:19.514786 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.514765 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln"] Oct 31 13:16:19.519733 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.519714 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-xd2ln"] Oct 31 13:16:19.536858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.536837 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq"] Oct 31 13:16:19.542797 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:19.542777 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-5dngq"] Oct 31 13:16:20.359031 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:20.358997 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:16:20.359329 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:20.359066 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:16:20.475903 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:20.475867 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" event={"ID":"ae57b233-eccd-45cf-a8ec-93f681c4c042","Type":"ContainerStarted","Data":"7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20"} Oct 31 13:16:20.476102 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:20.475926 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:20.503671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:20.503623 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" podStartSLOduration=1.589223007 podStartE2EDuration="15.503609799s" totalImagesPullingTime="13.914386792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:20.502139469 +0000 UTC m=+997.347305887" watchObservedRunningTime="2026-10-31 13:16:20.503609799 +0000 UTC m=+997.348776216" Oct 31 13:16:21.290150 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:21.290117 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b015433d-b413-49a1-a515-f903370ba0e0" path="/var/lib/kubelet/pods/b015433d-b413-49a1-a515-f903370ba0e0/volumes" Oct 31 13:16:21.290580 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:21.290566 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" path="/var/lib/kubelet/pods/b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40/volumes" Oct 31 13:16:21.290961 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:21.290949 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" path="/var/lib/kubelet/pods/cbe40c82-7a02-4fa3-8987-3f88fdf3f60f/volumes" Oct 31 13:16:21.291672 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:21.291656 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" path="/var/lib/kubelet/pods/d6150f29-b21c-4142-8b99-7fb0af958d0d/volumes" Oct 31 13:16:21.484994 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:21.484969 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:16:23.833438 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:23.833400 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:16:26.165923 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:26.165889 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv"] Oct 31 13:16:26.166379 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:26.166083 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" containerID="cri-o://24c4b6923d4d42555c1aeeff1e525e48666458f4a9c45a6b9d8844c967554634" gracePeriod=30 Oct 31 13:16:26.167683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:26.167653 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn"] Oct 31 13:16:26.167908 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:26.167876 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" containerID="cri-o://ea395df10199b8348bb4861a6cf9bce03d01fae5924ccdec6f29b0077faf1db4" gracePeriod=30 Oct 31 13:16:26.169958 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:26.169931 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:16:26.181540 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:26.181498 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp"] Oct 31 13:16:26.181732 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:26.181698 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerName="istio-proxy" containerID="cri-o://04ba6391519ea3d7d03cbd11f1e01999258e531c60f8bd84d0de8f4529b5ff03" gracePeriod=30 Oct 31 13:16:28.524092 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:28.524057 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:28.524503 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:28.524115 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:30.644826 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:30.644791 4542 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:16:30.645239 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:30.644866 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:16:31.562194 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.561621 4542 generic.go:356] "Generic (PLEG): container finished" podID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerID="04ba6391519ea3d7d03cbd11f1e01999258e531c60f8bd84d0de8f4529b5ff03" exitCode=0 Oct 31 13:16:31.562194 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.561712 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" event={"ID":"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846","Type":"ContainerDied","Data":"04ba6391519ea3d7d03cbd11f1e01999258e531c60f8bd84d0de8f4529b5ff03"} Oct 31 13:16:31.564328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.563842 4542 generic.go:356] "Generic (PLEG): container finished" podID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerID="24c4b6923d4d42555c1aeeff1e525e48666458f4a9c45a6b9d8844c967554634" exitCode=0 Oct 31 13:16:31.564328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.563958 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" event={"ID":"b098adeb-3a2f-479b-a72c-93bb60f4bba9","Type":"ContainerDied","Data":"24c4b6923d4d42555c1aeeff1e525e48666458f4a9c45a6b9d8844c967554634"} Oct 31 13:16:31.566286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.566127 4542 generic.go:356] "Generic (PLEG): container finished" podID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerID="ea395df10199b8348bb4861a6cf9bce03d01fae5924ccdec6f29b0077faf1db4" exitCode=0 Oct 31 13:16:31.566286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.566152 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" event={"ID":"707ba835-82ff-49ae-b982-4f75ecce68ab","Type":"ContainerDied","Data":"ea395df10199b8348bb4861a6cf9bce03d01fae5924ccdec6f29b0077faf1db4"} Oct 31 13:16:31.566286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.566184 4542 scope.go:122] "RemoveContainer" containerID="dd96b181b7fc7c5e8aed552811fff513274df19cfe4c640981ee5d438d104d9d" Oct 31 13:16:31.574985 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.574753 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:16:31.587106 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.579125 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:16:31.602517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.602493 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:16:31.690359 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690327 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-envoy\") pod \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690383 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-certs\") pod \"707ba835-82ff-49ae-b982-4f75ecce68ab\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690418 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-envoy\") pod \"707ba835-82ff-49ae-b982-4f75ecce68ab\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690450 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-socket\") pod \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690506 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-envoy\") pod \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690542 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-kube-api-access-l8wh9\" (UniqueName: \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-kube-api-access-l8wh9\") pod \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690582 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-credential-socket\") pod \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690645 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707ba835-82ff-49ae-b982-4f75ecce68ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707ba835-82ff-49ae-b982-4f75ecce68ab-istiod-ca-cert\") pod \"707ba835-82ff-49ae-b982-4f75ecce68ab\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690689 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-podinfo\") pod \"707ba835-82ff-49ae-b982-4f75ecce68ab\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690728 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-token\" (UniqueName: \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-token\") pod \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690760 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-kube-api-access-sk4h4\" (UniqueName: \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-kube-api-access-sk4h4\") pod \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690807 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-socket\") pod \"707ba835-82ff-49ae-b982-4f75ecce68ab\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " Oct 31 13:16:31.690858 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690843 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-data\") pod \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690874 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-token\" (UniqueName: \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-token\") pod \"707ba835-82ff-49ae-b982-4f75ecce68ab\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690910 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-kube-api-access-qlmvl\" (UniqueName: \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-kube-api-access-qlmvl\") pod \"707ba835-82ff-49ae-b982-4f75ecce68ab\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690937 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-podinfo\") pod \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.690977 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-data\") pod \"707ba835-82ff-49ae-b982-4f75ecce68ab\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.691008 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-data\") pod \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.691051 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-podinfo\") pod \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.691086 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istiod-ca-cert\") pod \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.691114 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istiod-ca-cert\") pod \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.691144 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-certs\") pod \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.691180 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-credential-socket\") pod \"707ba835-82ff-49ae-b982-4f75ecce68ab\" (UID: \"707ba835-82ff-49ae-b982-4f75ecce68ab\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.691215 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-token\" (UniqueName: \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-token\") pod \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.691266 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-credential-socket\") pod \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.691312 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-certs\") pod \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\" (UID: \"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846\") " Oct 31 13:16:31.691467 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.691340 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-socket\") pod \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\" (UID: \"b098adeb-3a2f-479b-a72c-93bb60f4bba9\") " Oct 31 13:16:31.692218 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.692192 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-data" pod "b098adeb-3a2f-479b-a72c-93bb60f4bba9" (UID: "b098adeb-3a2f-479b-a72c-93bb60f4bba9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.693670 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.692608 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-socket" pod "b098adeb-3a2f-479b-a72c-93bb60f4bba9" (UID: "b098adeb-3a2f-479b-a72c-93bb60f4bba9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.693670 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.692865 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istiod-ca-cert" pod "b098adeb-3a2f-479b-a72c-93bb60f4bba9" (UID: "b098adeb-3a2f-479b-a72c-93bb60f4bba9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:31.693670 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.693062 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-credential-socket" pod "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" (UID: "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.693670 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.693199 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-certs" pod "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" (UID: "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.693670 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.693321 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-credential-socket" pod "b098adeb-3a2f-479b-a72c-93bb60f4bba9" (UID: "b098adeb-3a2f-479b-a72c-93bb60f4bba9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.693670 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.693448 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-certs" pod "707ba835-82ff-49ae-b982-4f75ecce68ab" (UID: "707ba835-82ff-49ae-b982-4f75ecce68ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.695457 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.694776 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-socket" pod "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" (UID: "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.695457 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.695142 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707ba835-82ff-49ae-b982-4f75ecce68ab-istiod-ca-cert" pod "707ba835-82ff-49ae-b982-4f75ecce68ab" (UID: "707ba835-82ff-49ae-b982-4f75ecce68ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:31.695948 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.695721 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istiod-ca-cert" pod "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" (UID: "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:31.696178 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.696054 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-certs" pod "b098adeb-3a2f-479b-a72c-93bb60f4bba9" (UID: "b098adeb-3a2f-479b-a72c-93bb60f4bba9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.696458 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.696267 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-data" pod "707ba835-82ff-49ae-b982-4f75ecce68ab" (UID: "707ba835-82ff-49ae-b982-4f75ecce68ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.697609 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.697097 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-envoy" pod "707ba835-82ff-49ae-b982-4f75ecce68ab" (UID: "707ba835-82ff-49ae-b982-4f75ecce68ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.697609 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.697340 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-credential-socket" pod "707ba835-82ff-49ae-b982-4f75ecce68ab" (UID: "707ba835-82ff-49ae-b982-4f75ecce68ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.698323 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.698049 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-data" pod "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" (UID: "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.698574 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.698546 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-socket" pod "707ba835-82ff-49ae-b982-4f75ecce68ab" (UID: "707ba835-82ff-49ae-b982-4f75ecce68ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.698659 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.698644 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-kube-api-access-qlmvl" pod "707ba835-82ff-49ae-b982-4f75ecce68ab" (UID: "707ba835-82ff-49ae-b982-4f75ecce68ab"). InnerVolumeSpecName "kube-api-access-qlmvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:31.704155 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.704137 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-kube-api-access-l8wh9" pod "b098adeb-3a2f-479b-a72c-93bb60f4bba9" (UID: "b098adeb-3a2f-479b-a72c-93bb60f4bba9"). InnerVolumeSpecName "kube-api-access-l8wh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:31.704807 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.704773 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-podinfo" pod "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" (UID: "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:31.704807 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.704785 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-kube-api-access-sk4h4" pod "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" (UID: "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846"). InnerVolumeSpecName "kube-api-access-sk4h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:31.704974 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.704954 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-token" pod "b098adeb-3a2f-479b-a72c-93bb60f4bba9" (UID: "b098adeb-3a2f-479b-a72c-93bb60f4bba9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:31.705126 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.705096 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-podinfo" pod "707ba835-82ff-49ae-b982-4f75ecce68ab" (UID: "707ba835-82ff-49ae-b982-4f75ecce68ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:31.705126 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.705106 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-envoy" pod "b098adeb-3a2f-479b-a72c-93bb60f4bba9" (UID: "b098adeb-3a2f-479b-a72c-93bb60f4bba9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.705328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.705303 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-token" pod "707ba835-82ff-49ae-b982-4f75ecce68ab" (UID: "707ba835-82ff-49ae-b982-4f75ecce68ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:31.705513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.705470 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-envoy" pod "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" (UID: "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:16:31.705597 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.705551 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-token" pod "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" (UID: "09eedd25-a0ab-4d2e-85d1-62ee9ad2a846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:31.706718 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.706697 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-podinfo" pod "b098adeb-3a2f-479b-a72c-93bb60f4bba9" (UID: "b098adeb-3a2f-479b-a72c-93bb60f4bba9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795252 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795293 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707ba835-82ff-49ae-b982-4f75ecce68ab-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795309 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795325 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795340 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk4h4\" (UniqueName: \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-kube-api-access-sk4h4\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795357 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795371 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795384 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795396 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlmvl\" (UniqueName: \"kubernetes.io/projected/707ba835-82ff-49ae-b982-4f75ecce68ab-kube-api-access-qlmvl\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795411 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795425 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795440 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795453 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795467 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795501 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795514 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795528 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795542 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795557 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795572 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795585 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795598 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795614 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795628 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707ba835-82ff-49ae-b982-4f75ecce68ab-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795642 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795656 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b098adeb-3a2f-479b-a72c-93bb60f4bba9-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:31.795787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:31.795670 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8wh9\" (UniqueName: \"kubernetes.io/projected/b098adeb-3a2f-479b-a72c-93bb60f4bba9-kube-api-access-l8wh9\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:32.610000 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.609946 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" event={"ID":"09eedd25-a0ab-4d2e-85d1-62ee9ad2a846","Type":"ContainerDied","Data":"d0640bf81351515b5d3e1dca25de3a26af0f4238b03582b9eef5ddf848a0726b"} Oct 31 13:16:32.610303 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.610267 4542 scope.go:122] "RemoveContainer" containerID="04ba6391519ea3d7d03cbd11f1e01999258e531c60f8bd84d0de8f4529b5ff03" Oct 31 13:16:32.633338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.614765 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp" Oct 31 13:16:32.633338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.631707 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" event={"ID":"b098adeb-3a2f-479b-a72c-93bb60f4bba9","Type":"ContainerDied","Data":"17fe5fcd3b1ff01eb074cee65466b22249ca0dc5aa601b01d020d684a8764b43"} Oct 31 13:16:32.633338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.631824 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv" Oct 31 13:16:32.633338 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.633096 4542 scope.go:122] "RemoveContainer" containerID="24c4b6923d4d42555c1aeeff1e525e48666458f4a9c45a6b9d8844c967554634" Oct 31 13:16:32.655466 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.648943 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" event={"ID":"707ba835-82ff-49ae-b982-4f75ecce68ab","Type":"ContainerDied","Data":"50eff0dc565bf461f344786c2b07fc5bddd34801dd32cf06888f0f1b4ca2dbe9"} Oct 31 13:16:32.655466 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.649041 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn" Oct 31 13:16:32.677184 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.677113 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp"] Oct 31 13:16:32.687021 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.686995 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-5kbnp"] Oct 31 13:16:32.718055 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.718025 4542 scope.go:122] "RemoveContainer" containerID="ea395df10199b8348bb4861a6cf9bce03d01fae5924ccdec6f29b0077faf1db4" Oct 31 13:16:32.845687 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.845605 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv"] Oct 31 13:16:32.873141 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.873007 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-7vhrv"] Oct 31 13:16:32.916708 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.916673 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vl2sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:16:32.916997 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.916964 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vl2sm" podUID="ce755ffc-b2d4-45c2-9054-6022e18db453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:16:32.953665 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.953635 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn"] Oct 31 13:16:32.976745 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:32.976711 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-gmvvn"] Oct 31 13:16:33.334762 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:33.334709 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" path="/var/lib/kubelet/pods/09eedd25-a0ab-4d2e-85d1-62ee9ad2a846/volumes" Oct 31 13:16:33.347893 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:33.347855 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" path="/var/lib/kubelet/pods/707ba835-82ff-49ae-b982-4f75ecce68ab/volumes" Oct 31 13:16:33.362504 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:16:33.353027 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" path="/var/lib/kubelet/pods/b098adeb-3a2f-479b-a72c-93bb60f4bba9/volumes" Oct 31 13:16:53.833113 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:16:53.833069 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:17:23.831607 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:17:23.831571 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:17:36.932963 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:17:36.932924 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:17:43.626755 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:17:43.626728 4542 scope.go:122] "RemoveContainer" containerID="b73524eec74a6cf109978f6d10642882ffa26f38a8be05e8c2113525b3f0c2a9" Oct 31 13:17:43.634973 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:17:43.634949 4542 scope.go:122] "RemoveContainer" containerID="6490e9e5fb0ed1838508baadbfc058e6bab65c52b4717ac54083c77634d387b5" Oct 31 13:17:53.831068 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:17:53.831029 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:17:55.229153 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:17:55.229115 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:18:00.430758 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:18:00.430715 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:18:16.338968 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:18:16.338935 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:18:21.416611 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:18:21.416573 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xvwjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:18:21.417254 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:18:21.417220 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" podUID="3defcda2-f67c-4057-b3c4-ec130bbd4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:18:23.831501 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:18:23.831450 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:18:29.232074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:18:29.231997 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:18:53.831739 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:18:53.831705 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:18:53.926050 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:18:53.926013 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:19:04.190772 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:19:04.190737 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:19:09.290706 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:19:09.290678 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:19:23.831371 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:19:23.831337 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:19:24.837154 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:19:24.837113 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:19:29.032502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:19:29.032446 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:19:43.220350 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:19:43.220315 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:19:43.662780 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:19:43.662748 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:19:43.667718 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:19:43.667699 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:19:47.315078 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:19:47.315046 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:19:50.195512 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:19:50.194596 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nq8qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:19:50.195512 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:19:50.194669 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" podUID="9ec44fe2-a89f-4ae6-8f6f-7220557b19b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:19:53.833494 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:19:53.833434 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:20:04.104309 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:04.104232 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:20:08.056500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:08.056280 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:20:21.826489 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:21.826444 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:20:23.837865 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:20:23.837815 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:20:32.532086 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:32.532047 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:20:35.186500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:35.182570 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nq8qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:20:35.186500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:35.182638 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nq8qc" podUID="9ec44fe2-a89f-4ae6-8f6f-7220557b19b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:20:37.237840 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:37.237801 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:20:43.669380 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:43.669351 4542 scope.go:122] "RemoveContainer" containerID="0191a9bf51a7e7763c4841c141ea03c319fa07e05577e9d48ff8b9a4b9fb23e3" Oct 31 13:20:53.830977 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:20:53.830930 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:20:54.517673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:54.517640 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:20:57.027499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:57.027448 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc"] Oct 31 13:20:57.030961 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:57.030931 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ec18d26-58klc"] Oct 31 13:20:57.290371 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:57.290296 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9922f9-afbe-4381-85a8-7c77c251cfba" path="/var/lib/kubelet/pods/6d9922f9-afbe-4381-85a8-7c77c251cfba/volumes" Oct 31 13:20:58.507522 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:20:58.507471 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:21:07.523099 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:07.523065 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:21:23.831996 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:21:23.831946 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:21:25.893854 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:25.893824 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:21:36.013530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.013433 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:21:36.282979 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.282890 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw"] Oct 31 13:21:36.283139 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.283098 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" podUID="ae57b233-eccd-45cf-a8ec-93f681c4c042" containerName="discovery" containerID="cri-o://7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20" gracePeriod=30 Oct 31 13:21:36.575239 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.575206 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:21:36.578853 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.578830 4542 generic.go:356] "Generic (PLEG): container finished" podID="ae57b233-eccd-45cf-a8ec-93f681c4c042" containerID="7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20" exitCode=0 Oct 31 13:21:36.578958 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.578868 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" Oct 31 13:21:36.578958 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.578920 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" event={"ID":"ae57b233-eccd-45cf-a8ec-93f681c4c042","Type":"ContainerDied","Data":"7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20"} Oct 31 13:21:36.579055 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.578972 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" event={"ID":"ae57b233-eccd-45cf-a8ec-93f681c4c042","Type":"ContainerDied","Data":"3d53b00f21128adb0f5c818512c3f99f15832072bd76b3a078eb0410d27d2e64"} Oct 31 13:21:36.579055 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.578999 4542 scope.go:122] "RemoveContainer" containerID="7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20" Oct 31 13:21:36.587207 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.587192 4542 scope.go:122] "RemoveContainer" containerID="7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20" Oct 31 13:21:36.587468 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:21:36.587446 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20\": container with ID starting with 7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20 not found: ID does not exist" containerID="7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20" Oct 31 13:21:36.587573 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.587490 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20"} err="failed to get container status \"7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20\": rpc error: code = NotFound desc = could not find container \"7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20\": container with ID starting with 7739ebea0b43d1a99344f1a43dfde70cce2b7c036d3f3ac810d0380a4ada5e20 not found: ID does not exist" Oct 31 13:21:36.688560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.688534 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-cacerts\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-cacerts\") pod \"ae57b233-eccd-45cf-a8ec-93f681c4c042\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " Oct 31 13:21:36.688702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.688578 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-kube-api-access-2lq2p\" (UniqueName: \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-kube-api-access-2lq2p\") pod \"ae57b233-eccd-45cf-a8ec-93f681c4c042\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " Oct 31 13:21:36.688702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.688608 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-kubeconfig\") pod \"ae57b233-eccd-45cf-a8ec-93f681c4c042\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " Oct 31 13:21:36.688702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.688627 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-dns-cert\") pod \"ae57b233-eccd-45cf-a8ec-93f681c4c042\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " Oct 31 13:21:36.688702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.688673 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-token\" (UniqueName: \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-token\") pod \"ae57b233-eccd-45cf-a8ec-93f681c4c042\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " Oct 31 13:21:36.688702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.688697 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-ca-configmap\") pod \"ae57b233-eccd-45cf-a8ec-93f681c4c042\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " Oct 31 13:21:36.688993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.688799 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae57b233-eccd-45cf-a8ec-93f681c4c042-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae57b233-eccd-45cf-a8ec-93f681c4c042-local-certs\") pod \"ae57b233-eccd-45cf-a8ec-93f681c4c042\" (UID: \"ae57b233-eccd-45cf-a8ec-93f681c4c042\") " Oct 31 13:21:36.689171 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.689143 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-ca-configmap" pod "ae57b233-eccd-45cf-a8ec-93f681c4c042" (UID: "ae57b233-eccd-45cf-a8ec-93f681c4c042"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:36.689300 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.689240 4542 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-ca-configmap\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:36.690976 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.690940 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-token" pod "ae57b233-eccd-45cf-a8ec-93f681c4c042" (UID: "ae57b233-eccd-45cf-a8ec-93f681c4c042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:36.691229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.691197 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae57b233-eccd-45cf-a8ec-93f681c4c042-local-certs" pod "ae57b233-eccd-45cf-a8ec-93f681c4c042" (UID: "ae57b233-eccd-45cf-a8ec-93f681c4c042"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:36.691308 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.691240 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-kubeconfig" pod "ae57b233-eccd-45cf-a8ec-93f681c4c042" (UID: "ae57b233-eccd-45cf-a8ec-93f681c4c042"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:21:36.691308 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.691240 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-kube-api-access-2lq2p" pod "ae57b233-eccd-45cf-a8ec-93f681c4c042" (UID: "ae57b233-eccd-45cf-a8ec-93f681c4c042"). InnerVolumeSpecName "kube-api-access-2lq2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:36.691415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.691319 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-cacerts" pod "ae57b233-eccd-45cf-a8ec-93f681c4c042" (UID: "ae57b233-eccd-45cf-a8ec-93f681c4c042"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:21:36.691415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.691376 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-dns-cert" pod "ae57b233-eccd-45cf-a8ec-93f681c4c042" (UID: "ae57b233-eccd-45cf-a8ec-93f681c4c042"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:21:36.790684 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.790657 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lq2p\" (UniqueName: \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-kube-api-access-2lq2p\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:36.790684 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.790682 4542 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-kubeconfig\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:36.790829 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.790692 4542 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-csr-dns-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:36.790829 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.790700 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae57b233-eccd-45cf-a8ec-93f681c4c042-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:36.790829 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.790709 4542 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae57b233-eccd-45cf-a8ec-93f681c4c042-local-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:36.790829 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.790718 4542 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ae57b233-eccd-45cf-a8ec-93f681c4c042-cacerts\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:36.902860 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.902830 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw"] Oct 31 13:21:36.908321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:36.908296 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw"] Oct 31 13:21:37.291090 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:37.291014 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae57b233-eccd-45cf-a8ec-93f681c4c042" path="/var/lib/kubelet/pods/ae57b233-eccd-45cf-a8ec-93f681c4c042/volumes" Oct 31 13:21:41.484866 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:41.484828 4542 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jqfzw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:21:41.485279 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:41.484881 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jqfzw" podUID="ae57b233-eccd-45cf-a8ec-93f681c4c042" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.77:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:21:52.503010 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:52.502979 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:21:54.012229 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:21:54.012185 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:21:57.119563 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:21:57.119525 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:22:02.645056 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645024 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r"] Oct 31 13:22:02.645549 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645529 4542 cpu_manager.go:414] "containerMap: removing container" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerName="istio-proxy" Oct 31 13:22:02.645608 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645553 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerName="istio-proxy" Oct 31 13:22:02.645608 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645570 4542 cpu_manager.go:414] "containerMap: removing container" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" Oct 31 13:22:02.645608 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645579 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" Oct 31 13:22:02.645608 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645602 4542 cpu_manager.go:414] "containerMap: removing container" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645612 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645631 4542 cpu_manager.go:414] "containerMap: removing container" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645638 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645650 4542 cpu_manager.go:414] "containerMap: removing container" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645657 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645666 4542 cpu_manager.go:414] "containerMap: removing container" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645671 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645680 4542 cpu_manager.go:414] "containerMap: removing container" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645684 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645695 4542 cpu_manager.go:414] "containerMap: removing container" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645699 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645704 4542 cpu_manager.go:414] "containerMap: removing container" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645709 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645715 4542 cpu_manager.go:414] "containerMap: removing container" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645719 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645725 4542 cpu_manager.go:414] "containerMap: removing container" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645729 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645737 4542 cpu_manager.go:414] "containerMap: removing container" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645741 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645748 4542 cpu_manager.go:414] "containerMap: removing container" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645753 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" Oct 31 13:22:02.645749 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645759 4542 cpu_manager.go:414] "containerMap: removing container" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645763 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645771 4542 cpu_manager.go:414] "containerMap: removing container" podUID="ae57b233-eccd-45cf-a8ec-93f681c4c042" containerName="discovery" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645775 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae57b233-eccd-45cf-a8ec-93f681c4c042" containerName="discovery" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645783 4542 cpu_manager.go:414] "containerMap: removing container" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645787 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645793 4542 cpu_manager.go:414] "containerMap: removing container" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645797 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645804 4542 cpu_manager.go:414] "containerMap: removing container" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645807 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645816 4542 cpu_manager.go:414] "containerMap: removing container" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645820 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645828 4542 cpu_manager.go:414] "containerMap: removing container" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645832 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645839 4542 cpu_manager.go:414] "containerMap: removing container" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645843 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645849 4542 cpu_manager.go:414] "containerMap: removing container" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645853 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645924 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="f067868a-a184-4ef4-9f5c-7de384be46ae" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645933 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="2604e553-ae64-4fdf-ba10-1a639e5157f0" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645939 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae57b233-eccd-45cf-a8ec-93f681c4c042" containerName="discovery" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645945 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645950 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645956 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="e02d8eed-d690-4d65-981a-03b5a4331852" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645964 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8e330f-01c1-435e-b684-56a1ed9208b7" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645969 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="b015433d-b413-49a1-a515-f903370ba0e0" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645974 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d85d750-b72f-4664-b3b2-1a0f802a6d81" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645982 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48e0675-ee4f-4089-adcd-8c6eff556d2d" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.645991 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646000 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="09eedd25-a0ab-4d2e-85d1-62ee9ad2a846" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646007 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="b098adeb-3a2f-479b-a72c-93bb60f4bba9" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646015 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d057adb-f72c-4327-b979-aa61fdf98068" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646021 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b70fcb-394c-4c0d-baa8-d0c8323c0723" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646027 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646031 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7eb1ab-7542-44aa-a93d-d3186844631a" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646036 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c6d6ca-4bc9-4b35-8133-fa2f7637ed40" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646042 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48b8206-4423-4c72-8ade-a37075d55f97" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646047 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabf6be0-70ac-4a23-bce4-23a0047c0cfc" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646052 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ddbe95-4c3e-494d-9257-accc94089b63" containerName="istio-proxy" Oct 31 13:22:02.646403 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.646057 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6150f29-b21c-4142-8b99-7fb0af958d0d" containerName="istio-proxy" Oct 31 13:22:02.650696 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.650680 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.657299 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.656887 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-25d309-openshift-default-dockercfg-hs55k\"" Oct 31 13:22:02.663418 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.663396 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r"] Oct 31 13:22:02.705814 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.705784 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-token\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.705943 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.705820 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e79f36-92c4-48d7-9a02-897b9077218a-istiod-ca-cert\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.705943 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.705853 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-socket\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.705943 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.705925 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-certs\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.706055 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.705956 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-envoy\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.706055 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.705993 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-data\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.706123 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.706051 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-credential-socket\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.706160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.706131 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-podinfo\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.706194 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.706183 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2shl\" (UniqueName: \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-kube-api-access-r2shl\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807126 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807101 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-socket\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807283 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807131 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-certs\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807283 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807146 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-envoy\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807283 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807161 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-data\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807283 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807176 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-credential-socket\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807283 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807196 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-podinfo\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807283 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807264 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2shl\" (UniqueName: \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-kube-api-access-r2shl\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807368 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-token\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807391 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e79f36-92c4-48d7-9a02-897b9077218a-istiod-ca-cert\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807574 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-certs\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807605 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-credential-socket\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807810 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807709 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-data\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.807810 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.807735 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-socket\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.808053 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.808034 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e79f36-92c4-48d7-9a02-897b9077218a-istiod-ca-cert\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.809596 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.809577 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-envoy\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.809664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.809648 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-podinfo\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.821509 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.821466 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2shl\" (UniqueName: \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-kube-api-access-r2shl\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.821675 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.821652 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-token\") pod \"e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:02.969248 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:02.969152 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:03.251292 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:03.251217 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r"] Oct 31 13:22:03.254465 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:22:03.254437 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e79f36_92c4_48d7_9a02_897b9077218a.slice/crio-e728cabf163f4c25582b188dc3575f780b90b57d550d9806d417d9b1dd4f0514 WatchSource:0}: Error finding container e728cabf163f4c25582b188dc3575f780b90b57d550d9806d417d9b1dd4f0514: Status 404 returned error can't find the container with id e728cabf163f4c25582b188dc3575f780b90b57d550d9806d417d9b1dd4f0514 Oct 31 13:22:03.256735 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:03.256699 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:22:03.256829 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:03.256782 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:22:03.256885 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:03.256828 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:22:03.776819 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:03.776785 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" event={"ID":"c3e79f36-92c4-48d7-9a02-897b9077218a","Type":"ContainerStarted","Data":"296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a"} Oct 31 13:22:03.777226 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:03.776845 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" event={"ID":"c3e79f36-92c4-48d7-9a02-897b9077218a","Type":"ContainerStarted","Data":"e728cabf163f4c25582b188dc3575f780b90b57d550d9806d417d9b1dd4f0514"} Oct 31 13:22:03.803020 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:03.802931 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" podStartSLOduration=1.8029129 podStartE2EDuration="1.8029129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:03.801926558 +0000 UTC m=+1340.647093009" watchObservedRunningTime="2026-10-31 13:22:03.8029129 +0000 UTC m=+1340.648079319" Oct 31 13:22:03.969856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:03.969824 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:04.975150 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:04.975120 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:05.785505 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:05.785460 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:05.786491 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:05.786454 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:07.377695 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.377660 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk"] Oct 31 13:22:07.378160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.378083 4542 cpu_manager.go:414] "containerMap: removing container" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" Oct 31 13:22:07.378160 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.378100 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe40c82-7a02-4fa3-8987-3f88fdf3f60f" containerName="istio-proxy" Oct 31 13:22:07.378268 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.378207 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="707ba835-82ff-49ae-b982-4f75ecce68ab" containerName="istio-proxy" Oct 31 13:22:07.381504 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.381461 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:22:07.392243 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.392221 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-25d309\"" Oct 31 13:22:07.394235 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.394213 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk"] Oct 31 13:22:07.415220 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.415199 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-9656703a-dockercfg-5t4x2\"" Oct 31 13:22:07.419088 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.419064 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j"] Oct 31 13:22:07.422557 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.422540 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:22:07.440618 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.440494 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j"] Oct 31 13:22:07.454382 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.454357 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jds\" (UniqueName: \"kubernetes.io/projected/da301326-05e7-4d06-a842-b7a9a27b0c2a-kube-api-access-h6jds\") pod \"payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk\" (UID: \"da301326-05e7-4d06-a842-b7a9a27b0c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:22:07.454501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.454421 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da301326-05e7-4d06-a842-b7a9a27b0c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk\" (UID: \"da301326-05e7-4d06-a842-b7a9a27b0c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:22:07.555155 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.555102 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j\" (UID: \"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:22:07.555364 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.555164 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jds\" (UniqueName: \"kubernetes.io/projected/da301326-05e7-4d06-a842-b7a9a27b0c2a-kube-api-access-h6jds\") pod \"payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk\" (UID: \"da301326-05e7-4d06-a842-b7a9a27b0c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:22:07.555364 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.555304 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da301326-05e7-4d06-a842-b7a9a27b0c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk\" (UID: \"da301326-05e7-4d06-a842-b7a9a27b0c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:22:07.555364 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.555360 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhmk\" (UniqueName: \"kubernetes.io/projected/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-kube-api-access-zfhmk\") pod \"payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j\" (UID: \"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:22:07.557694 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.557663 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da301326-05e7-4d06-a842-b7a9a27b0c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk\" (UID: \"da301326-05e7-4d06-a842-b7a9a27b0c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:22:07.572138 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.572107 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jds\" (UniqueName: \"kubernetes.io/projected/da301326-05e7-4d06-a842-b7a9a27b0c2a-kube-api-access-h6jds\") pod \"payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk\" (UID: \"da301326-05e7-4d06-a842-b7a9a27b0c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:22:07.656198 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.656115 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhmk\" (UniqueName: \"kubernetes.io/projected/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-kube-api-access-zfhmk\") pod \"payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j\" (UID: \"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:22:07.656377 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.656238 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j\" (UID: \"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:22:07.656950 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.656930 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j\" (UID: \"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:22:07.674165 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.674134 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhmk\" (UniqueName: \"kubernetes.io/projected/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-kube-api-access-zfhmk\") pod \"payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j\" (UID: \"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:22:07.695037 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.695010 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:22:07.750096 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.750068 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:22:07.969501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:07.969456 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk"] Oct 31 13:22:07.971132 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:22:07.971108 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda301326_05e7_4d06_a842_b7a9a27b0c2a.slice/crio-362d1cf733b341cef2bbda947f2c78bcac8efa7decba3b5966c88ee77764a59c WatchSource:0}: Error finding container 362d1cf733b341cef2bbda947f2c78bcac8efa7decba3b5966c88ee77764a59c: Status 404 returned error can't find the container with id 362d1cf733b341cef2bbda947f2c78bcac8efa7decba3b5966c88ee77764a59c Oct 31 13:22:08.039534 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:08.039508 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j"] Oct 31 13:22:08.041804 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:22:08.041779 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ed17a4_1953_4b73_99b7_ccdf9b8bc9c6.slice/crio-ce57c416fccb600fb24e1d341f1761e5590d316d282915e7ea2f99288a55d016 WatchSource:0}: Error finding container ce57c416fccb600fb24e1d341f1761e5590d316d282915e7ea2f99288a55d016: Status 404 returned error can't find the container with id ce57c416fccb600fb24e1d341f1761e5590d316d282915e7ea2f99288a55d016 Oct 31 13:22:08.800647 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:08.800612 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" event={"ID":"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6","Type":"ContainerStarted","Data":"b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6"} Oct 31 13:22:08.801100 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:08.800674 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:22:08.801100 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:08.800699 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" event={"ID":"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6","Type":"ContainerStarted","Data":"ce57c416fccb600fb24e1d341f1761e5590d316d282915e7ea2f99288a55d016"} Oct 31 13:22:08.802279 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:08.802254 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" event={"ID":"da301326-05e7-4d06-a842-b7a9a27b0c2a","Type":"ContainerStarted","Data":"b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb"} Oct 31 13:22:08.802368 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:08.802297 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" event={"ID":"da301326-05e7-4d06-a842-b7a9a27b0c2a","Type":"ContainerStarted","Data":"362d1cf733b341cef2bbda947f2c78bcac8efa7decba3b5966c88ee77764a59c"} Oct 31 13:22:08.802368 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:08.802359 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:22:08.826344 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:08.826296 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" podStartSLOduration=1.826280227 podStartE2EDuration="1.826280227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:08.82380547 +0000 UTC m=+1345.668971889" watchObservedRunningTime="2026-10-31 13:22:08.826280227 +0000 UTC m=+1345.671446649" Oct 31 13:22:08.844517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:08.844461 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" podStartSLOduration=1.844448083 podStartE2EDuration="1.844448083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:08.843373971 +0000 UTC m=+1345.688540390" watchObservedRunningTime="2026-10-31 13:22:08.844448083 +0000 UTC m=+1345.689614501" Oct 31 13:22:10.108958 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:10.108916 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:22:19.809918 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:19.809894 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:22:19.810284 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:19.810046 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:22:19.858999 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:19.858924 4542 generic.go:356] "Generic (PLEG): container finished" podID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerID="296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a" exitCode=0 Oct 31 13:22:19.859154 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:19.858991 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" event={"ID":"c3e79f36-92c4-48d7-9a02-897b9077218a","Type":"ContainerDied","Data":"296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a"} Oct 31 13:22:19.859404 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:19.859389 4542 scope.go:122] "RemoveContainer" containerID="296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a" Oct 31 13:22:19.860797 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:19.860760 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:22:19.860949 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:19.860840 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:22:19.860949 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:19.860885 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:22:20.785743 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:20.785705 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:20.870303 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:20.870252 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" event={"ID":"c3e79f36-92c4-48d7-9a02-897b9077218a","Type":"ContainerStarted","Data":"fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce"} Oct 31 13:22:20.970038 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:20.969999 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:20.971255 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:20.971233 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:22:20.971365 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:20.971285 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 13:22:21.970785 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:21.970725 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:22:21.970785 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:21.970780 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 13:22:22.969985 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:22.969943 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:22:22.970172 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:22.970012 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 13:22:23.831011 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:22:23.830975 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:22:23.970073 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:23.970034 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:22:23.970256 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:23.970101 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 13:22:24.513226 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:24.513194 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:22:24.969735 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:24.969699 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:22:24.970150 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:24.969766 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 13:22:26.345571 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:26.345543 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:26.346673 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:26.346650 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:26.347212 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:26.347197 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:22:42.100313 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:42.100280 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:22:43.699553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:43.699523 4542 scope.go:122] "RemoveContainer" containerID="90c18c3f49486cd76005f2cc169a43ddd8640fc1f17c82f74a4fb32c139d679a" Oct 31 13:22:43.708309 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:22:43.708289 4542 scope.go:122] "RemoveContainer" containerID="88efee26a9e941d97ed20f998e5f163ce2d5e19306643266893a15ecb3b9b89a" Oct 31 13:22:53.834517 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:22:53.834464 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:23:07.190502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:07.190389 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:23:16.587118 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:16.587085 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j"] Oct 31 13:23:16.603638 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:16.603512 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk"] Oct 31 13:23:19.955452 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:19.955415 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:23:21.587994 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:21.587938 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" podUID="11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6" containerName="payload-processing" containerID="cri-o://b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6" gracePeriod=25 Oct 31 13:23:21.604441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:21.604409 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" podUID="da301326-05e7-4d06-a842-b7a9a27b0c2a" containerName="payload-pre-processing" containerID="cri-o://b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb" gracePeriod=25 Oct 31 13:23:23.831945 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:23:23.831907 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:23:25.186649 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:25.186619 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r"] Oct 31 13:23:25.187066 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:25.186854 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" containerID="cri-o://fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce" gracePeriod=30 Oct 31 13:23:26.348461 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:26.348429 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:23:26.348888 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:26.348515 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:23:29.808982 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:29.808942 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" podUID="11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:23:29.809399 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:29.808942 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" podUID="da301326-05e7-4d06-a842-b7a9a27b0c2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:23:30.443289 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.443266 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:23:30.510176 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510115 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e79f36-92c4-48d7-9a02-897b9077218a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e79f36-92c4-48d7-9a02-897b9077218a-istiod-ca-cert\") pod \"c3e79f36-92c4-48d7-9a02-897b9077218a\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " Oct 31 13:23:30.510321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510180 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-socket\") pod \"c3e79f36-92c4-48d7-9a02-897b9077218a\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " Oct 31 13:23:30.510321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510219 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-kube-api-access-r2shl\" (UniqueName: \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-kube-api-access-r2shl\") pod \"c3e79f36-92c4-48d7-9a02-897b9077218a\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " Oct 31 13:23:30.510321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510240 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-certs\") pod \"c3e79f36-92c4-48d7-9a02-897b9077218a\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " Oct 31 13:23:30.510321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510263 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-data\") pod \"c3e79f36-92c4-48d7-9a02-897b9077218a\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " Oct 31 13:23:30.510321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510287 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-token\" (UniqueName: \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-token\") pod \"c3e79f36-92c4-48d7-9a02-897b9077218a\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " Oct 31 13:23:30.510321 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510316 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-podinfo\") pod \"c3e79f36-92c4-48d7-9a02-897b9077218a\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " Oct 31 13:23:30.510664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510346 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-credential-socket\") pod \"c3e79f36-92c4-48d7-9a02-897b9077218a\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " Oct 31 13:23:30.510664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510388 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-envoy\") pod \"c3e79f36-92c4-48d7-9a02-897b9077218a\" (UID: \"c3e79f36-92c4-48d7-9a02-897b9077218a\") " Oct 31 13:23:30.510664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510418 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-socket" pod "c3e79f36-92c4-48d7-9a02-897b9077218a" (UID: "c3e79f36-92c4-48d7-9a02-897b9077218a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:30.510664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510618 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-certs" pod "c3e79f36-92c4-48d7-9a02-897b9077218a" (UID: "c3e79f36-92c4-48d7-9a02-897b9077218a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:30.510664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510643 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-data" pod "c3e79f36-92c4-48d7-9a02-897b9077218a" (UID: "c3e79f36-92c4-48d7-9a02-897b9077218a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:30.510664 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510645 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e79f36-92c4-48d7-9a02-897b9077218a-istiod-ca-cert" pod "c3e79f36-92c4-48d7-9a02-897b9077218a" (UID: "c3e79f36-92c4-48d7-9a02-897b9077218a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:30.510959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510778 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-credential-socket" pod "c3e79f36-92c4-48d7-9a02-897b9077218a" (UID: "c3e79f36-92c4-48d7-9a02-897b9077218a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:30.510959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510892 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e79f36-92c4-48d7-9a02-897b9077218a-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:30.510959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510908 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:30.510959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510918 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:30.510959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510927 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:30.510959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.510939 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:30.512398 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.512372 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-podinfo" pod "c3e79f36-92c4-48d7-9a02-897b9077218a" (UID: "c3e79f36-92c4-48d7-9a02-897b9077218a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:23:30.512569 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.512507 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-envoy" pod "c3e79f36-92c4-48d7-9a02-897b9077218a" (UID: "c3e79f36-92c4-48d7-9a02-897b9077218a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:30.512855 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.512830 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-token" pod "c3e79f36-92c4-48d7-9a02-897b9077218a" (UID: "c3e79f36-92c4-48d7-9a02-897b9077218a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:30.512855 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.512844 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-kube-api-access-r2shl" pod "c3e79f36-92c4-48d7-9a02-897b9077218a" (UID: "c3e79f36-92c4-48d7-9a02-897b9077218a"). InnerVolumeSpecName "kube-api-access-r2shl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:30.611562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.611536 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2shl\" (UniqueName: \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-kube-api-access-r2shl\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:30.611562 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.611561 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:30.611697 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.611570 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:30.611697 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:30.611580 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e79f36-92c4-48d7-9a02-897b9077218a-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:31.208193 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.208157 4542 generic.go:356] "Generic (PLEG): container finished" podID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerID="fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce" exitCode=0 Oct 31 13:23:31.208738 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.208206 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" Oct 31 13:23:31.208738 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.208242 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" event={"ID":"c3e79f36-92c4-48d7-9a02-897b9077218a","Type":"ContainerDied","Data":"fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce"} Oct 31 13:23:31.208738 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.208314 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r" event={"ID":"c3e79f36-92c4-48d7-9a02-897b9077218a","Type":"ContainerDied","Data":"e728cabf163f4c25582b188dc3575f780b90b57d550d9806d417d9b1dd4f0514"} Oct 31 13:23:31.208738 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.208351 4542 scope.go:122] "RemoveContainer" containerID="fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce" Oct 31 13:23:31.217671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.217655 4542 scope.go:122] "RemoveContainer" containerID="296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a" Oct 31 13:23:31.226008 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.225992 4542 scope.go:122] "RemoveContainer" containerID="fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce" Oct 31 13:23:31.226246 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:23:31.226231 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce\": container with ID starting with fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce not found: ID does not exist" containerID="fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce" Oct 31 13:23:31.226300 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.226255 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce"} err="failed to get container status \"fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce\": rpc error: code = NotFound desc = could not find container \"fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce\": container with ID starting with fa205cc1f91d816ac3a12d81256ac67abe0311c4c84ae1717222952738ef20ce not found: ID does not exist" Oct 31 13:23:31.226300 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.226271 4542 scope.go:122] "RemoveContainer" containerID="296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a" Oct 31 13:23:31.226514 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:23:31.226496 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a\": container with ID starting with 296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a not found: ID does not exist" containerID="296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a" Oct 31 13:23:31.226572 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.226524 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a"} err="failed to get container status \"296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a\": rpc error: code = NotFound desc = could not find container \"296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a\": container with ID starting with 296267d93b0e5573a8aa39c604c43a18e7484cd9f464ba516bea66027a254c5a not found: ID does not exist" Oct 31 13:23:31.233368 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.233346 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r"] Oct 31 13:23:31.238243 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.238221 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-25d309-openshift-default-cc845854f-h7l9r"] Oct 31 13:23:31.290712 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:31.290690 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" path="/var/lib/kubelet/pods/c3e79f36-92c4-48d7-9a02-897b9077218a/volumes" Oct 31 13:23:38.023601 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:38.023565 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:23:39.821502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:39.820562 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" podUID="da301326-05e7-4d06-a842-b7a9a27b0c2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:23:39.821502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:39.821051 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" podUID="11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:23:41.853221 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:41.853164 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:23:41.874223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:41.874188 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:23:41.944959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:41.944919 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da301326-05e7-4d06-a842-b7a9a27b0c2a-kube-api-access-h6jds\" (UniqueName: \"kubernetes.io/projected/da301326-05e7-4d06-a842-b7a9a27b0c2a-kube-api-access-h6jds\") pod \"da301326-05e7-4d06-a842-b7a9a27b0c2a\" (UID: \"da301326-05e7-4d06-a842-b7a9a27b0c2a\") " Oct 31 13:23:41.945117 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:41.944991 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-plugins-config-volume\") pod \"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6\" (UID: \"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6\") " Oct 31 13:23:41.945117 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:41.945057 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da301326-05e7-4d06-a842-b7a9a27b0c2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da301326-05e7-4d06-a842-b7a9a27b0c2a-plugins-config-volume\") pod \"da301326-05e7-4d06-a842-b7a9a27b0c2a\" (UID: \"da301326-05e7-4d06-a842-b7a9a27b0c2a\") " Oct 31 13:23:41.945117 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:41.945083 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-kube-api-access-zfhmk\" (UniqueName: \"kubernetes.io/projected/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-kube-api-access-zfhmk\") pod \"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6\" (UID: \"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6\") " Oct 31 13:23:41.945461 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:41.945432 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-plugins-config-volume" pod "11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6" (UID: "11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:41.945591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:41.945440 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da301326-05e7-4d06-a842-b7a9a27b0c2a-plugins-config-volume" pod "da301326-05e7-4d06-a842-b7a9a27b0c2a" (UID: "da301326-05e7-4d06-a842-b7a9a27b0c2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:41.947079 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:41.947057 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da301326-05e7-4d06-a842-b7a9a27b0c2a-kube-api-access-h6jds" pod "da301326-05e7-4d06-a842-b7a9a27b0c2a" (UID: "da301326-05e7-4d06-a842-b7a9a27b0c2a"). InnerVolumeSpecName "kube-api-access-h6jds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:41.947190 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:41.947168 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-kube-api-access-zfhmk" pod "11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6" (UID: "11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6"). InnerVolumeSpecName "kube-api-access-zfhmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:42.046870 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.046837 4542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da301326-05e7-4d06-a842-b7a9a27b0c2a-plugins-config-volume\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:42.046870 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.046863 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfhmk\" (UniqueName: \"kubernetes.io/projected/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-kube-api-access-zfhmk\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:42.046870 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.046872 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6jds\" (UniqueName: \"kubernetes.io/projected/da301326-05e7-4d06-a842-b7a9a27b0c2a-kube-api-access-h6jds\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:42.047080 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.046881 4542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6-plugins-config-volume\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:42.273289 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.273206 4542 generic.go:356] "Generic (PLEG): container finished" podID="da301326-05e7-4d06-a842-b7a9a27b0c2a" containerID="b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb" exitCode=0 Oct 31 13:23:42.273289 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.273252 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" Oct 31 13:23:42.273523 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.273295 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" event={"ID":"da301326-05e7-4d06-a842-b7a9a27b0c2a","Type":"ContainerDied","Data":"b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb"} Oct 31 13:23:42.273523 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.273359 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk" event={"ID":"da301326-05e7-4d06-a842-b7a9a27b0c2a","Type":"ContainerDied","Data":"362d1cf733b341cef2bbda947f2c78bcac8efa7decba3b5966c88ee77764a59c"} Oct 31 13:23:42.273523 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.273396 4542 scope.go:122] "RemoveContainer" containerID="b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb" Oct 31 13:23:42.274737 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.274715 4542 generic.go:356] "Generic (PLEG): container finished" podID="11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6" containerID="b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6" exitCode=0 Oct 31 13:23:42.274845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.274744 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" event={"ID":"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6","Type":"ContainerDied","Data":"b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6"} Oct 31 13:23:42.274845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.274765 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" Oct 31 13:23:42.274845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.274784 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j" event={"ID":"11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6","Type":"ContainerDied","Data":"ce57c416fccb600fb24e1d341f1761e5590d316d282915e7ea2f99288a55d016"} Oct 31 13:23:42.286582 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.284175 4542 scope.go:122] "RemoveContainer" containerID="b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb" Oct 31 13:23:42.286582 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:23:42.284702 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb\": container with ID starting with b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb not found: ID does not exist" containerID="b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb" Oct 31 13:23:42.286582 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.284779 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb"} err="failed to get container status \"b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb\": rpc error: code = NotFound desc = could not find container \"b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb\": container with ID starting with b7c9552be9abbb5b246911a22221a32c3cd6d281377f9268f8d671e00f09bedb not found: ID does not exist" Oct 31 13:23:42.286582 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.284818 4542 scope.go:122] "RemoveContainer" containerID="b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6" Oct 31 13:23:42.295184 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.295164 4542 scope.go:122] "RemoveContainer" containerID="b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6" Oct 31 13:23:42.295439 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:23:42.295420 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6\": container with ID starting with b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6 not found: ID does not exist" containerID="b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6" Oct 31 13:23:42.295511 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.295445 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6"} err="failed to get container status \"b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6\": rpc error: code = NotFound desc = could not find container \"b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6\": container with ID starting with b74d207eed1fdbae22e469dc06875aad73656617fc5f131ea2659ec56f0e11a6 not found: ID does not exist" Oct 31 13:23:42.304894 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.304875 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk"] Oct 31 13:23:42.309979 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.309960 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-25d309-7c7dccb6nk"] Oct 31 13:23:42.323431 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.323410 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j"] Oct 31 13:23:42.328080 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.328061 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-25d309-675dc756b4rv6j"] Oct 31 13:23:42.797170 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:42.797138 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:23:43.289512 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:43.289459 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6" path="/var/lib/kubelet/pods/11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6/volumes" Oct 31 13:23:43.290035 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:23:43.290013 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da301326-05e7-4d06-a842-b7a9a27b0c2a" path="/var/lib/kubelet/pods/da301326-05e7-4d06-a842-b7a9a27b0c2a/volumes" Oct 31 13:23:53.831437 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:23:53.831404 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:24:23.831843 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:24:23.831806 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:24:43.693437 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:24:43.693406 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:24:43.698162 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:24:43.698142 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:24:53.834055 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:24:53.834017 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:25:08.419413 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:08.419378 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:25:23.648166 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:23.648131 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:25:23.831648 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:25:23.831612 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:25:27.467500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:27.466519 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:25:42.623062 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:42.623023 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:25:43.688358 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688327 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn"] Oct 31 13:25:43.688750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688734 4542 cpu_manager.go:414] "containerMap: removing container" podUID="da301326-05e7-4d06-a842-b7a9a27b0c2a" containerName="payload-pre-processing" Oct 31 13:25:43.688750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688746 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da301326-05e7-4d06-a842-b7a9a27b0c2a" containerName="payload-pre-processing" Oct 31 13:25:43.688827 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688760 4542 cpu_manager.go:414] "containerMap: removing container" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" Oct 31 13:25:43.688827 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688765 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" Oct 31 13:25:43.688827 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688785 4542 cpu_manager.go:414] "containerMap: removing container" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" Oct 31 13:25:43.688827 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688789 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" Oct 31 13:25:43.688827 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688803 4542 cpu_manager.go:414] "containerMap: removing container" podUID="11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6" containerName="payload-processing" Oct 31 13:25:43.688827 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688810 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6" containerName="payload-processing" Oct 31 13:25:43.689023 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688869 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" Oct 31 13:25:43.689023 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688881 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ed17a4-1953-4b73-99b7-ccdf9b8bc9c6" containerName="payload-processing" Oct 31 13:25:43.689023 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.688895 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="da301326-05e7-4d06-a842-b7a9a27b0c2a" containerName="payload-pre-processing" Oct 31 13:25:43.692081 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.692064 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.705458 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.705434 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn"] Oct 31 13:25:43.710690 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.710649 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2c64dd-openshift-default-dockercfg-2f7kz\"" Oct 31 13:25:43.721362 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.721336 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-envoy\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.721517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.721371 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4z9\" (UniqueName: \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-kube-api-access-7p4z9\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.721517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.721394 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0afedcc-1420-48ab-9833-75fe6e9498b1-istiod-ca-cert\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.721517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.721503 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-data\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.721713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.721601 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-certs\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.721713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.721661 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-socket\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.721822 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.721780 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-token\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.721951 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.721834 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-credential-socket\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.721951 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.721936 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-podinfo\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.823584 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.823544 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-podinfo\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.824856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.823613 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-envoy\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.824856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.823647 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4z9\" (UniqueName: \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-kube-api-access-7p4z9\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.824856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.823676 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0afedcc-1420-48ab-9833-75fe6e9498b1-istiod-ca-cert\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.824856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.823730 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-data\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.824856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.823760 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-certs\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.824856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.824268 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-socket\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.824856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.824349 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-token\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.824856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.824382 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-credential-socket\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.825581 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.824964 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0afedcc-1420-48ab-9833-75fe6e9498b1-istiod-ca-cert\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.825719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.825694 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-credential-socket\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.825906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.825881 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-certs\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.826042 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.826018 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-socket\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.826264 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.826242 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-data\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.828660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.828430 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-envoy\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.828660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.828597 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-podinfo\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.841273 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.841248 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4z9\" (UniqueName: \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-kube-api-access-7p4z9\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:43.842212 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:43.842191 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-token\") pod \"e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:44.005151 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:44.005067 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:44.289175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:44.289150 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn"] Oct 31 13:25:44.291423 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:25:44.291396 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0afedcc_1420_48ab_9833_75fe6e9498b1.slice/crio-72060ca1bf1bddb5e56c2edb552d0f234bd1e316641676d881c4c03abd69353b WatchSource:0}: Error finding container 72060ca1bf1bddb5e56c2edb552d0f234bd1e316641676d881c4c03abd69353b: Status 404 returned error can't find the container with id 72060ca1bf1bddb5e56c2edb552d0f234bd1e316641676d881c4c03abd69353b Oct 31 13:25:44.293498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:44.293449 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:25:44.293580 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:44.293529 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:25:44.293580 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:44.293563 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:25:44.865816 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:44.865780 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" event={"ID":"c0afedcc-1420-48ab-9833-75fe6e9498b1","Type":"ContainerStarted","Data":"e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d"} Oct 31 13:25:44.866288 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:44.865843 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" event={"ID":"c0afedcc-1420-48ab-9833-75fe6e9498b1","Type":"ContainerStarted","Data":"72060ca1bf1bddb5e56c2edb552d0f234bd1e316641676d881c4c03abd69353b"} Oct 31 13:25:44.892203 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:44.892146 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" podStartSLOduration=1.892129699 podStartE2EDuration="1.892129699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:44.8917013 +0000 UTC m=+1561.736867718" watchObservedRunningTime="2026-10-31 13:25:44.892129699 +0000 UTC m=+1561.737296117" Oct 31 13:25:45.006233 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:45.006079 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:45.012702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:45.012660 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:45.870341 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:45.870310 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:45.871312 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:45.871292 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:25:48.501369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.501338 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4"] Oct 31 13:25:48.501813 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.501792 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e79f36-92c4-48d7-9a02-897b9077218a" containerName="istio-proxy" Oct 31 13:25:48.504811 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.504794 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:25:48.510422 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.510402 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-69ab7f90-dockercfg-hz5pv\"" Oct 31 13:25:48.510643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.510627 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2c64dd\"" Oct 31 13:25:48.532470 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.532448 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4"] Oct 31 13:25:48.548330 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.548301 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85"] Oct 31 13:25:48.551867 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.551845 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:25:48.562760 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.560430 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85"] Oct 31 13:25:48.568801 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.568775 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrfth\" (UniqueName: \"kubernetes.io/projected/29d55d56-56e8-46ca-a45c-07b05b937d77-kube-api-access-xrfth\") pod \"payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4\" (UID: \"29d55d56-56e8-46ca-a45c-07b05b937d77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:25:48.568925 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.568895 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d55d56-56e8-46ca-a45c-07b05b937d77-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4\" (UID: \"29d55d56-56e8-46ca-a45c-07b05b937d77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:25:48.670101 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.670073 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85\" (UID: \"07a95b77-1dfe-48ac-a7ab-8bd0cd055254\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:25:48.670300 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.670126 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d55d56-56e8-46ca-a45c-07b05b937d77-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4\" (UID: \"29d55d56-56e8-46ca-a45c-07b05b937d77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:25:48.670300 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.670195 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzt88\" (UniqueName: \"kubernetes.io/projected/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-kube-api-access-kzt88\") pod \"payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85\" (UID: \"07a95b77-1dfe-48ac-a7ab-8bd0cd055254\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:25:48.670300 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.670232 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrfth\" (UniqueName: \"kubernetes.io/projected/29d55d56-56e8-46ca-a45c-07b05b937d77-kube-api-access-xrfth\") pod \"payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4\" (UID: \"29d55d56-56e8-46ca-a45c-07b05b937d77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:25:48.670913 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.670886 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d55d56-56e8-46ca-a45c-07b05b937d77-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4\" (UID: \"29d55d56-56e8-46ca-a45c-07b05b937d77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:25:48.684180 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.684146 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrfth\" (UniqueName: \"kubernetes.io/projected/29d55d56-56e8-46ca-a45c-07b05b937d77-kube-api-access-xrfth\") pod \"payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4\" (UID: \"29d55d56-56e8-46ca-a45c-07b05b937d77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:25:48.771627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.771560 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85\" (UID: \"07a95b77-1dfe-48ac-a7ab-8bd0cd055254\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:25:48.771627 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.771622 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt88\" (UniqueName: \"kubernetes.io/projected/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-kube-api-access-kzt88\") pod \"payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85\" (UID: \"07a95b77-1dfe-48ac-a7ab-8bd0cd055254\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:25:48.772214 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.772194 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85\" (UID: \"07a95b77-1dfe-48ac-a7ab-8bd0cd055254\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:25:48.784959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.784934 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzt88\" (UniqueName: \"kubernetes.io/projected/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-kube-api-access-kzt88\") pod \"payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85\" (UID: \"07a95b77-1dfe-48ac-a7ab-8bd0cd055254\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:25:48.819265 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.819240 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:25:48.867813 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:48.867767 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:25:49.126954 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:49.126922 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4"] Oct 31 13:25:49.131882 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:25:49.131850 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d55d56_56e8_46ca_a45c_07b05b937d77.slice/crio-2235dbea73d1f0d5836347936de821e3316b952a743a12f0139678fc5e8b4d6f WatchSource:0}: Error finding container 2235dbea73d1f0d5836347936de821e3316b952a743a12f0139678fc5e8b4d6f: Status 404 returned error can't find the container with id 2235dbea73d1f0d5836347936de821e3316b952a743a12f0139678fc5e8b4d6f Oct 31 13:25:49.178263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:49.178238 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85"] Oct 31 13:25:49.181218 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:25:49.181193 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a95b77_1dfe_48ac_a7ab_8bd0cd055254.slice/crio-029ac2b5a7f516a2fb0e911c7e3d406af895eb3863ff4dc1570e503b8ce3850d WatchSource:0}: Error finding container 029ac2b5a7f516a2fb0e911c7e3d406af895eb3863ff4dc1570e503b8ce3850d: Status 404 returned error can't find the container with id 029ac2b5a7f516a2fb0e911c7e3d406af895eb3863ff4dc1570e503b8ce3850d Oct 31 13:25:49.891621 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:49.891582 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" event={"ID":"07a95b77-1dfe-48ac-a7ab-8bd0cd055254","Type":"ContainerStarted","Data":"013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763"} Oct 31 13:25:49.892028 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:49.891645 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" event={"ID":"07a95b77-1dfe-48ac-a7ab-8bd0cd055254","Type":"ContainerStarted","Data":"029ac2b5a7f516a2fb0e911c7e3d406af895eb3863ff4dc1570e503b8ce3850d"} Oct 31 13:25:49.892028 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:49.891671 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:25:49.893143 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:49.893113 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" event={"ID":"29d55d56-56e8-46ca-a45c-07b05b937d77","Type":"ContainerStarted","Data":"638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a"} Oct 31 13:25:49.893255 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:49.893159 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" event={"ID":"29d55d56-56e8-46ca-a45c-07b05b937d77","Type":"ContainerStarted","Data":"2235dbea73d1f0d5836347936de821e3316b952a743a12f0139678fc5e8b4d6f"} Oct 31 13:25:49.893255 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:49.893194 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:25:49.911961 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:49.911922 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" podStartSLOduration=1.911911001 podStartE2EDuration="1.911911001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:49.910633309 +0000 UTC m=+1566.755799722" watchObservedRunningTime="2026-10-31 13:25:49.911911001 +0000 UTC m=+1566.757077409" Oct 31 13:25:49.929926 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:49.929883 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" podStartSLOduration=1.929870534 podStartE2EDuration="1.929870534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:49.929301694 +0000 UTC m=+1566.774468113" watchObservedRunningTime="2026-10-31 13:25:49.929870534 +0000 UTC m=+1566.775036953" Oct 31 13:25:51.142713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:25:51.142682 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:25:53.832747 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:25:53.832714 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:26:00.871868 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:00.871834 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 13:26:00.872247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:00.871915 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 13:26:00.902088 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:00.901690 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:26:00.902609 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:00.902584 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:26:00.958778 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:00.958702 4542 generic.go:356] "Generic (PLEG): container finished" podID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerID="e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d" exitCode=0 Oct 31 13:26:00.958778 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:00.958743 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" event={"ID":"c0afedcc-1420-48ab-9833-75fe6e9498b1","Type":"ContainerDied","Data":"e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d"} Oct 31 13:26:00.959201 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:00.959181 4542 scope.go:122] "RemoveContainer" containerID="e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d" Oct 31 13:26:00.960499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:00.960444 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:26:00.960588 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:00.960528 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:26:00.960588 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:00.960562 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:26:01.969215 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:01.969177 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" event={"ID":"c0afedcc-1420-48ab-9833-75fe6e9498b1","Type":"ContainerStarted","Data":"2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95"} Oct 31 13:26:02.005378 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:02.005343 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:26:02.005554 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:02.005536 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 13:26:02.005613 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:02.005578 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 13:26:02.522278 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:02.522240 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:26:03.005610 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:03.005574 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 13:26:03.006002 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:03.005625 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 13:26:04.005552 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:04.005520 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 13:26:04.005740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:04.005574 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 13:26:05.005530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:05.005492 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 13:26:05.005746 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:05.005571 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 13:26:06.005471 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:06.005440 4542 patch_prober.go:29] interesting pod/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 13:26:06.005668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:06.005512 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 13:26:07.463568 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:07.463524 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:26:07.464432 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:07.464406 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:26:07.464935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:07.464916 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:26:16.013541 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:16.013507 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:26:23.832351 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:26:23.832310 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:26:40.389425 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:40.389390 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:26:50.326991 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:50.326951 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85"] Oct 31 13:26:50.333750 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:50.333724 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4"] Oct 31 13:26:53.242713 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:53.242682 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:26:53.831547 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:26:53.831506 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:26:55.327231 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:55.327193 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" podUID="07a95b77-1dfe-48ac-a7ab-8bd0cd055254" containerName="payload-processing" containerID="cri-o://013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763" gracePeriod=25 Oct 31 13:26:55.334987 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:55.334955 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" podUID="29d55d56-56e8-46ca-a45c-07b05b937d77" containerName="payload-pre-processing" containerID="cri-o://638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a" gracePeriod=25 Oct 31 13:26:59.425272 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:59.425242 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn"] Oct 31 13:26:59.425732 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:26:59.425453 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" containerID="cri-o://2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95" gracePeriod=30 Oct 31 13:27:00.899433 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:00.899388 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" podUID="07a95b77-1dfe-48ac-a7ab-8bd0cd055254" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:27:00.899433 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:00.899404 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" podUID="29d55d56-56e8-46ca-a45c-07b05b937d77" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:27:03.967131 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:03.967083 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt"] Oct 31 13:27:03.970903 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:03.970881 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:03.986182 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:03.986160 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-19627735-kserve-self-signed-certs\"" Oct 31 13:27:03.992254 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:03.992233 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt"] Oct 31 13:27:04.059051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.059021 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-dshm\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.059189 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.059072 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-model-cache\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.059189 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.059088 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd1ca627-9a11-48d1-ac66-9110898aa642-tls-certs\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.059189 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.059170 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-tmp-dir\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.059368 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.059320 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-home\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.059437 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.059420 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr8sx\" (UniqueName: \"kubernetes.io/projected/bd1ca627-9a11-48d1-ac66-9110898aa642-kube-api-access-jr8sx\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.160915 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.160881 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-dshm\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.161071 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.160932 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-model-cache\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.161071 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.160951 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd1ca627-9a11-48d1-ac66-9110898aa642-tls-certs\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.161071 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.160967 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-tmp-dir\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.161071 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.160997 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-home\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.161071 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.161038 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr8sx\" (UniqueName: \"kubernetes.io/projected/bd1ca627-9a11-48d1-ac66-9110898aa642-kube-api-access-jr8sx\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.161399 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.161378 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-model-cache\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.161465 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.161381 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-tmp-dir\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.161465 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.161419 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-home\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.162989 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.162972 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-dshm\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.163356 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.163333 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd1ca627-9a11-48d1-ac66-9110898aa642-tls-certs\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.181454 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.181428 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr8sx\" (UniqueName: \"kubernetes.io/projected/bd1ca627-9a11-48d1-ac66-9110898aa642-kube-api-access-jr8sx\") pod \"e2e-same-id-a-19627735-kserve-577767d477-dv5gt\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.281671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.281594 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:04.559702 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.559556 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt"] Oct 31 13:27:04.561871 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:27:04.561846 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd1ca627_9a11_48d1_ac66_9110898aa642.slice/crio-560744b92c633944ca60d1a92d6477f04167288ee9ca5dd6e82d432927913da4 WatchSource:0}: Error finding container 560744b92c633944ca60d1a92d6477f04167288ee9ca5dd6e82d432927913da4: Status 404 returned error can't find the container with id 560744b92c633944ca60d1a92d6477f04167288ee9ca5dd6e82d432927913da4 Oct 31 13:27:04.671499 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.671447 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:27:04.767185 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767154 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-envoy\") pod \"c0afedcc-1420-48ab-9833-75fe6e9498b1\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " Oct 31 13:27:04.767334 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767200 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-token\" (UniqueName: \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-token\") pod \"c0afedcc-1420-48ab-9833-75fe6e9498b1\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " Oct 31 13:27:04.767334 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767263 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-podinfo\") pod \"c0afedcc-1420-48ab-9833-75fe6e9498b1\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " Oct 31 13:27:04.767334 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767322 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-credential-socket\") pod \"c0afedcc-1420-48ab-9833-75fe6e9498b1\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " Oct 31 13:27:04.767550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767387 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-certs\") pod \"c0afedcc-1420-48ab-9833-75fe6e9498b1\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " Oct 31 13:27:04.767550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767417 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-kube-api-access-7p4z9\" (UniqueName: \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-kube-api-access-7p4z9\") pod \"c0afedcc-1420-48ab-9833-75fe6e9498b1\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " Oct 31 13:27:04.767550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767463 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-data\") pod \"c0afedcc-1420-48ab-9833-75fe6e9498b1\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " Oct 31 13:27:04.767550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767538 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-socket\") pod \"c0afedcc-1420-48ab-9833-75fe6e9498b1\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " Oct 31 13:27:04.767745 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767595 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0afedcc-1420-48ab-9833-75fe6e9498b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0afedcc-1420-48ab-9833-75fe6e9498b1-istiod-ca-cert\") pod \"c0afedcc-1420-48ab-9833-75fe6e9498b1\" (UID: \"c0afedcc-1420-48ab-9833-75fe6e9498b1\") " Oct 31 13:27:04.767745 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767703 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-credential-socket" pod "c0afedcc-1420-48ab-9833-75fe6e9498b1" (UID: "c0afedcc-1420-48ab-9833-75fe6e9498b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:27:04.767745 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767716 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-certs" pod "c0afedcc-1420-48ab-9833-75fe6e9498b1" (UID: "c0afedcc-1420-48ab-9833-75fe6e9498b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:27:04.767888 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767830 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-data" pod "c0afedcc-1420-48ab-9833-75fe6e9498b1" (UID: "c0afedcc-1420-48ab-9833-75fe6e9498b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:27:04.767963 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.767924 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-socket" pod "c0afedcc-1420-48ab-9833-75fe6e9498b1" (UID: "c0afedcc-1420-48ab-9833-75fe6e9498b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:27:04.768031 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.768011 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0afedcc-1420-48ab-9833-75fe6e9498b1-istiod-ca-cert" pod "c0afedcc-1420-48ab-9833-75fe6e9498b1" (UID: "c0afedcc-1420-48ab-9833-75fe6e9498b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:04.768229 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.768207 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:04.768286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.768237 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:04.768286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.768252 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:04.768286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.768267 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:04.768286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.768282 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0afedcc-1420-48ab-9833-75fe6e9498b1-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:04.769454 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.769432 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-token" pod "c0afedcc-1420-48ab-9833-75fe6e9498b1" (UID: "c0afedcc-1420-48ab-9833-75fe6e9498b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:04.769761 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.769741 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-kube-api-access-7p4z9" pod "c0afedcc-1420-48ab-9833-75fe6e9498b1" (UID: "c0afedcc-1420-48ab-9833-75fe6e9498b1"). InnerVolumeSpecName "kube-api-access-7p4z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:04.769838 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.769781 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-podinfo" pod "c0afedcc-1420-48ab-9833-75fe6e9498b1" (UID: "c0afedcc-1420-48ab-9833-75fe6e9498b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:27:04.769838 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.769784 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-envoy" pod "c0afedcc-1420-48ab-9833-75fe6e9498b1" (UID: "c0afedcc-1420-48ab-9833-75fe6e9498b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:27:04.869659 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.869635 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p4z9\" (UniqueName: \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-kube-api-access-7p4z9\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:04.869659 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.869660 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:04.869826 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.869673 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:04.869826 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:04.869684 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0afedcc-1420-48ab-9833-75fe6e9498b1-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:05.275519 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.275462 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" event={"ID":"bd1ca627-9a11-48d1-ac66-9110898aa642","Type":"ContainerStarted","Data":"66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288"} Oct 31 13:27:05.275937 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.275528 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" event={"ID":"bd1ca627-9a11-48d1-ac66-9110898aa642","Type":"ContainerStarted","Data":"560744b92c633944ca60d1a92d6477f04167288ee9ca5dd6e82d432927913da4"} Oct 31 13:27:05.277175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.277146 4542 generic.go:356] "Generic (PLEG): container finished" podID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerID="2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95" exitCode=0 Oct 31 13:27:05.277310 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.277188 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" event={"ID":"c0afedcc-1420-48ab-9833-75fe6e9498b1","Type":"ContainerDied","Data":"2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95"} Oct 31 13:27:05.277310 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.277262 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" event={"ID":"c0afedcc-1420-48ab-9833-75fe6e9498b1","Type":"ContainerDied","Data":"72060ca1bf1bddb5e56c2edb552d0f234bd1e316641676d881c4c03abd69353b"} Oct 31 13:27:05.277310 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.277282 4542 scope.go:122] "RemoveContainer" containerID="2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95" Oct 31 13:27:05.277441 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.277189 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn" Oct 31 13:27:05.282435 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.282416 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:05.286596 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.286347 4542 scope.go:122] "RemoveContainer" containerID="e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d" Oct 31 13:27:05.295192 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.295176 4542 scope.go:122] "RemoveContainer" containerID="2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95" Oct 31 13:27:05.295559 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:27:05.295536 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95\": container with ID starting with 2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95 not found: ID does not exist" containerID="2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95" Oct 31 13:27:05.295654 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.295565 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95"} err="failed to get container status \"2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95\": rpc error: code = NotFound desc = could not find container \"2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95\": container with ID starting with 2cd13693456924051329a3da51818ff6d17468d50d5b71668e5fb2cfe50bad95 not found: ID does not exist" Oct 31 13:27:05.295654 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.295589 4542 scope.go:122] "RemoveContainer" containerID="e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d" Oct 31 13:27:05.295887 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:27:05.295866 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d\": container with ID starting with e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d not found: ID does not exist" containerID="e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d" Oct 31 13:27:05.295952 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.295893 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d"} err="failed to get container status \"e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d\": rpc error: code = NotFound desc = could not find container \"e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d\": container with ID starting with e858b735a7633f7d5b5eec30339fa8e38692de0ba0f1901c468254c8b615c32d not found: ID does not exist" Oct 31 13:27:05.296401 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.296361 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" podStartSLOduration=2.296349846 podStartE2EDuration="2.296349846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:05.295393406 +0000 UTC m=+1642.140559823" watchObservedRunningTime="2026-10-31 13:27:05.296349846 +0000 UTC m=+1642.141516264" Oct 31 13:27:05.311370 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.311348 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn"] Oct 31 13:27:05.317743 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.317724 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2c64dd-openshift-default-55bd85d4d5x2bkn"] Oct 31 13:27:05.626973 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:05.626944 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:27:07.302500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:07.302322 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" path="/var/lib/kubelet/pods/c0afedcc-1420-48ab-9833-75fe6e9498b1/volumes" Oct 31 13:27:10.899316 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:10.899279 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" podUID="07a95b77-1dfe-48ac-a7ab-8bd0cd055254" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:27:10.899748 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:10.899279 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" podUID="29d55d56-56e8-46ca-a45c-07b05b937d77" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:27:14.282066 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:14.282032 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:14.294461 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:14.294431 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:14.362513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:14.362465 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:27:15.583685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.583626 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:27:15.612305 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.612281 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:27:15.707588 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.707560 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-kube-api-access-kzt88\" (UniqueName: \"kubernetes.io/projected/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-kube-api-access-kzt88\") pod \"07a95b77-1dfe-48ac-a7ab-8bd0cd055254\" (UID: \"07a95b77-1dfe-48ac-a7ab-8bd0cd055254\") " Oct 31 13:27:15.707588 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.707595 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d55d56-56e8-46ca-a45c-07b05b937d77-kube-api-access-xrfth\" (UniqueName: \"kubernetes.io/projected/29d55d56-56e8-46ca-a45c-07b05b937d77-kube-api-access-xrfth\") pod \"29d55d56-56e8-46ca-a45c-07b05b937d77\" (UID: \"29d55d56-56e8-46ca-a45c-07b05b937d77\") " Oct 31 13:27:15.707795 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.707755 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d55d56-56e8-46ca-a45c-07b05b937d77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d55d56-56e8-46ca-a45c-07b05b937d77-plugins-config-volume\") pod \"29d55d56-56e8-46ca-a45c-07b05b937d77\" (UID: \"29d55d56-56e8-46ca-a45c-07b05b937d77\") " Oct 31 13:27:15.707795 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.707784 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-plugins-config-volume\") pod \"07a95b77-1dfe-48ac-a7ab-8bd0cd055254\" (UID: \"07a95b77-1dfe-48ac-a7ab-8bd0cd055254\") " Oct 31 13:27:15.708165 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.708138 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d55d56-56e8-46ca-a45c-07b05b937d77-plugins-config-volume" pod "29d55d56-56e8-46ca-a45c-07b05b937d77" (UID: "29d55d56-56e8-46ca-a45c-07b05b937d77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:15.708252 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.708185 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-plugins-config-volume" pod "07a95b77-1dfe-48ac-a7ab-8bd0cd055254" (UID: "07a95b77-1dfe-48ac-a7ab-8bd0cd055254"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:15.709645 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.709628 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-kube-api-access-kzt88" pod "07a95b77-1dfe-48ac-a7ab-8bd0cd055254" (UID: "07a95b77-1dfe-48ac-a7ab-8bd0cd055254"). InnerVolumeSpecName "kube-api-access-kzt88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:15.709757 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.709738 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d55d56-56e8-46ca-a45c-07b05b937d77-kube-api-access-xrfth" pod "29d55d56-56e8-46ca-a45c-07b05b937d77" (UID: "29d55d56-56e8-46ca-a45c-07b05b937d77"). InnerVolumeSpecName "kube-api-access-xrfth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:15.809185 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.809149 4542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d55d56-56e8-46ca-a45c-07b05b937d77-plugins-config-volume\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:15.809185 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.809180 4542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-plugins-config-volume\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:15.809185 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.809189 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzt88\" (UniqueName: \"kubernetes.io/projected/07a95b77-1dfe-48ac-a7ab-8bd0cd055254-kube-api-access-kzt88\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:15.809405 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:15.809198 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrfth\" (UniqueName: \"kubernetes.io/projected/29d55d56-56e8-46ca-a45c-07b05b937d77-kube-api-access-xrfth\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:16.362049 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.362020 4542 generic.go:356] "Generic (PLEG): container finished" podID="29d55d56-56e8-46ca-a45c-07b05b937d77" containerID="638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a" exitCode=0 Oct 31 13:27:16.362224 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.362065 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" Oct 31 13:27:16.362224 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.362112 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" event={"ID":"29d55d56-56e8-46ca-a45c-07b05b937d77","Type":"ContainerDied","Data":"638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a"} Oct 31 13:27:16.362224 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.362166 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4" event={"ID":"29d55d56-56e8-46ca-a45c-07b05b937d77","Type":"ContainerDied","Data":"2235dbea73d1f0d5836347936de821e3316b952a743a12f0139678fc5e8b4d6f"} Oct 31 13:27:16.362224 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.362187 4542 scope.go:122] "RemoveContainer" containerID="638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a" Oct 31 13:27:16.363514 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.363494 4542 generic.go:356] "Generic (PLEG): container finished" podID="07a95b77-1dfe-48ac-a7ab-8bd0cd055254" containerID="013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763" exitCode=0 Oct 31 13:27:16.363604 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.363542 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" Oct 31 13:27:16.363604 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.363550 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" event={"ID":"07a95b77-1dfe-48ac-a7ab-8bd0cd055254","Type":"ContainerDied","Data":"013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763"} Oct 31 13:27:16.363604 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.363585 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85" event={"ID":"07a95b77-1dfe-48ac-a7ab-8bd0cd055254","Type":"ContainerDied","Data":"029ac2b5a7f516a2fb0e911c7e3d406af895eb3863ff4dc1570e503b8ce3850d"} Oct 31 13:27:16.370845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.370822 4542 scope.go:122] "RemoveContainer" containerID="638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a" Oct 31 13:27:16.371116 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:27:16.371095 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a\": container with ID starting with 638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a not found: ID does not exist" containerID="638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a" Oct 31 13:27:16.371195 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.371126 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a"} err="failed to get container status \"638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a\": rpc error: code = NotFound desc = could not find container \"638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a\": container with ID starting with 638a69f7e6edeb0fa7b3bdf07833473f278277bf1a1ce7e3e1e37414fba5844a not found: ID does not exist" Oct 31 13:27:16.371195 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.371148 4542 scope.go:122] "RemoveContainer" containerID="013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763" Oct 31 13:27:16.383439 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.383419 4542 scope.go:122] "RemoveContainer" containerID="013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763" Oct 31 13:27:16.383733 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:27:16.383714 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763\": container with ID starting with 013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763 not found: ID does not exist" containerID="013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763" Oct 31 13:27:16.383803 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.383739 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763"} err="failed to get container status \"013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763\": rpc error: code = NotFound desc = could not find container \"013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763\": container with ID starting with 013f6732767813d887bc50bada7520c92e8889b8a93b6c7e650b28e3fe4e1763 not found: ID does not exist" Oct 31 13:27:16.392205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.392183 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4"] Oct 31 13:27:16.397005 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.396983 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2c64dd-77469jcjw4"] Oct 31 13:27:16.409911 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.409890 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85"] Oct 31 13:27:16.415110 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:16.415091 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2c64dd-7b4d95897n9m85"] Oct 31 13:27:17.289924 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:17.289891 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a95b77-1dfe-48ac-a7ab-8bd0cd055254" path="/var/lib/kubelet/pods/07a95b77-1dfe-48ac-a7ab-8bd0cd055254/volumes" Oct 31 13:27:17.290339 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:17.290312 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d55d56-56e8-46ca-a45c-07b05b937d77" path="/var/lib/kubelet/pods/29d55d56-56e8-46ca-a45c-07b05b937d77/volumes" Oct 31 13:27:23.832164 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:27:23.832126 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:27:27.911665 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:27.911589 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:27:43.109472 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:43.109433 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:27:53.831212 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:27:53.831179 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:27:54.157351 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:27:54.157284 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:28:03.367997 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:03.367965 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt"] Oct 31 13:28:05.197283 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:05.197246 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:28:10.997607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:10.997552 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:28:18.387216 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.387158 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" podUID="bd1ca627-9a11-48d1-ac66-9110898aa642" containerName="main" containerID="cri-o://66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288" gracePeriod=45 Oct 31 13:28:18.650014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.649951 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:28:18.700357 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.700329 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd1ca627-9a11-48d1-ac66-9110898aa642-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd1ca627-9a11-48d1-ac66-9110898aa642-tls-certs\") pod \"bd1ca627-9a11-48d1-ac66-9110898aa642\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " Oct 31 13:28:18.700569 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.700388 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-home\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-home\") pod \"bd1ca627-9a11-48d1-ac66-9110898aa642\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " Oct 31 13:28:18.700569 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.700527 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-dshm\") pod \"bd1ca627-9a11-48d1-ac66-9110898aa642\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " Oct 31 13:28:18.700695 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.700586 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1ca627-9a11-48d1-ac66-9110898aa642-kube-api-access-jr8sx\" (UniqueName: \"kubernetes.io/projected/bd1ca627-9a11-48d1-ac66-9110898aa642-kube-api-access-jr8sx\") pod \"bd1ca627-9a11-48d1-ac66-9110898aa642\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " Oct 31 13:28:18.700695 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.700632 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-tmp-dir\") pod \"bd1ca627-9a11-48d1-ac66-9110898aa642\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " Oct 31 13:28:18.700695 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.700657 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-model-cache\") pod \"bd1ca627-9a11-48d1-ac66-9110898aa642\" (UID: \"bd1ca627-9a11-48d1-ac66-9110898aa642\") " Oct 31 13:28:18.700695 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.700680 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-home" pod "bd1ca627-9a11-48d1-ac66-9110898aa642" (UID: "bd1ca627-9a11-48d1-ac66-9110898aa642"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:28:18.700993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.700967 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-tmp-dir" pod "bd1ca627-9a11-48d1-ac66-9110898aa642" (UID: "bd1ca627-9a11-48d1-ac66-9110898aa642"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:28:18.701184 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.700990 4542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-home\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:18.701184 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.701166 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-model-cache" pod "bd1ca627-9a11-48d1-ac66-9110898aa642" (UID: "bd1ca627-9a11-48d1-ac66-9110898aa642"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:28:18.703357 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.703329 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd1ca627-9a11-48d1-ac66-9110898aa642-tls-certs" pod "bd1ca627-9a11-48d1-ac66-9110898aa642" (UID: "bd1ca627-9a11-48d1-ac66-9110898aa642"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:28:18.703455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.703381 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1ca627-9a11-48d1-ac66-9110898aa642-kube-api-access-jr8sx" pod "bd1ca627-9a11-48d1-ac66-9110898aa642" (UID: "bd1ca627-9a11-48d1-ac66-9110898aa642"). InnerVolumeSpecName "kube-api-access-jr8sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:18.703455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.703417 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-dshm" pod "bd1ca627-9a11-48d1-ac66-9110898aa642" (UID: "bd1ca627-9a11-48d1-ac66-9110898aa642"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:28:18.802247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.802199 4542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-dshm\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:18.802247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.802247 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr8sx\" (UniqueName: \"kubernetes.io/projected/bd1ca627-9a11-48d1-ac66-9110898aa642-kube-api-access-jr8sx\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:18.802247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.802258 4542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-tmp-dir\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:18.802553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.802268 4542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd1ca627-9a11-48d1-ac66-9110898aa642-model-cache\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:18.802553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.802280 4542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd1ca627-9a11-48d1-ac66-9110898aa642-tls-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:18.873501 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.873446 4542 generic.go:356] "Generic (PLEG): container finished" podID="bd1ca627-9a11-48d1-ac66-9110898aa642" containerID="66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288" exitCode=0 Oct 31 13:28:18.873660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.873501 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" Oct 31 13:28:18.873660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.873517 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" event={"ID":"bd1ca627-9a11-48d1-ac66-9110898aa642","Type":"ContainerDied","Data":"66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288"} Oct 31 13:28:18.873660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.873564 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt" event={"ID":"bd1ca627-9a11-48d1-ac66-9110898aa642","Type":"ContainerDied","Data":"560744b92c633944ca60d1a92d6477f04167288ee9ca5dd6e82d432927913da4"} Oct 31 13:28:18.873660 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.873582 4542 scope.go:122] "RemoveContainer" containerID="66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288" Oct 31 13:28:18.883072 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.883056 4542 scope.go:122] "RemoveContainer" containerID="66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288" Oct 31 13:28:18.883328 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:28:18.883308 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288\": container with ID starting with 66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288 not found: ID does not exist" containerID="66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288" Oct 31 13:28:18.883400 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.883339 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288"} err="failed to get container status \"66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288\": rpc error: code = NotFound desc = could not find container \"66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288\": container with ID starting with 66929253c3dd3b3cd6ddada5c271974c015caa5ab6de5a03cd53c7261d9d1288 not found: ID does not exist" Oct 31 13:28:18.897285 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.897252 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt"] Oct 31 13:28:18.904115 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:18.904065 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-19627735-kserve-577767d477-dv5gt"] Oct 31 13:28:19.289681 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:19.289603 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd1ca627-9a11-48d1-ac66-9110898aa642" path="/var/lib/kubelet/pods/bd1ca627-9a11-48d1-ac66-9110898aa642/volumes" Oct 31 13:28:19.430257 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:19.430225 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:28:21.417496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:21.416570 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xvwjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:28:21.417496 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:21.416684 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xvwjv" podUID="3defcda2-f67c-4057-b3c4-ec130bbd4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:21.536500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:21.535623 4542 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-5v9c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:28:21.536500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:21.535748 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-5v9c9" podUID="ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:23.996945 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:28:23.996905 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:28:24.008530 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:24.008494 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:28:39.919958 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:39.919929 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:28:48.671167 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.671132 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9"] Oct 31 13:28:48.671923 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.671902 4542 cpu_manager.go:414] "containerMap: removing container" podUID="bd1ca627-9a11-48d1-ac66-9110898aa642" containerName="main" Oct 31 13:28:48.672025 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.671925 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1ca627-9a11-48d1-ac66-9110898aa642" containerName="main" Oct 31 13:28:48.672025 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.671942 4542 cpu_manager.go:414] "containerMap: removing container" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" Oct 31 13:28:48.672025 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.671949 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" Oct 31 13:28:48.672025 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.671959 4542 cpu_manager.go:414] "containerMap: removing container" podUID="29d55d56-56e8-46ca-a45c-07b05b937d77" containerName="payload-pre-processing" Oct 31 13:28:48.672025 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.671967 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d55d56-56e8-46ca-a45c-07b05b937d77" containerName="payload-pre-processing" Oct 31 13:28:48.672025 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.672025 4542 cpu_manager.go:414] "containerMap: removing container" podUID="07a95b77-1dfe-48ac-a7ab-8bd0cd055254" containerName="payload-processing" Oct 31 13:28:48.672369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.672034 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a95b77-1dfe-48ac-a7ab-8bd0cd055254" containerName="payload-processing" Oct 31 13:28:48.672369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.672137 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a95b77-1dfe-48ac-a7ab-8bd0cd055254" containerName="payload-processing" Oct 31 13:28:48.672369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.672149 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" Oct 31 13:28:48.672369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.672159 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1ca627-9a11-48d1-ac66-9110898aa642" containerName="main" Oct 31 13:28:48.672369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.672170 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" Oct 31 13:28:48.672369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.672180 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d55d56-56e8-46ca-a45c-07b05b937d77" containerName="payload-pre-processing" Oct 31 13:28:48.676787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.676766 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.689258 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.689239 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-351178df-openshift-default-dockercfg-jfggh\"" Oct 31 13:28:48.704423 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.704396 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9"] Oct 31 13:28:48.812048 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.812012 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-certs\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.812269 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.812064 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-data\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.812269 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.812187 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-socket\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.812269 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.812232 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-envoy\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.812269 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.812261 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-podinfo\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.812437 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.812284 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-token\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.812437 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.812398 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ab6adf-84d6-4b87-975c-31ecabe6f630-istiod-ca-cert\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.812552 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.812457 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-credential-socket\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.812593 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.812547 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7z2\" (UniqueName: \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-kube-api-access-js7z2\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.913425 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.913386 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-certs\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.914008 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.913976 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-data\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.914008 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.914047 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-socket\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.914311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.914095 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-envoy\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.914311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.914127 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-podinfo\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.914311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.914149 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-token\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.914311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.914189 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ab6adf-84d6-4b87-975c-31ecabe6f630-istiod-ca-cert\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.914311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.914244 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-credential-socket\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.914311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.914306 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7z2\" (UniqueName: \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-kube-api-access-js7z2\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.914656 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.914357 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-data\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.915131 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.915102 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-socket\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.915596 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.915535 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-credential-socket\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.915711 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.915620 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ab6adf-84d6-4b87-975c-31ecabe6f630-istiod-ca-cert\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.917919 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.917891 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-podinfo\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.918735 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.918710 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-envoy\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.919666 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.919641 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-certs\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.933668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.933616 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-token\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.933747 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.933683 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7z2\" (UniqueName: \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-kube-api-access-js7z2\") pod \"e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:48.989997 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:48.989977 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:49.102906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.102874 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7"] Oct 31 13:28:49.103276 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.103263 4542 cpu_manager.go:414] "containerMap: removing container" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" Oct 31 13:28:49.103328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.103278 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0afedcc-1420-48ab-9833-75fe6e9498b1" containerName="istio-proxy" Oct 31 13:28:49.106745 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.106728 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.126564 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.126539 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7"] Oct 31 13:28:49.218376 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.218293 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxn2\" (UniqueName: \"kubernetes.io/projected/fa049032-484c-43c5-9440-7928760a0967-kube-api-access-jcxn2\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.218376 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.218340 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa049032-484c-43c5-9440-7928760a0967-openshift-service-ca\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.218376 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.218358 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa049032-484c-43c5-9440-7928760a0967-maas-api-tls\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.218683 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.218638 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa049032-484c-43c5-9440-7928760a0967-metrics-certs\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.275375 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.275340 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9"] Oct 31 13:28:49.277714 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:28:49.277686 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ab6adf_84d6_4b87_975c_31ecabe6f630.slice/crio-a71555022b802ad458fa97cc5631d0abac8bfe5e8b7ccd01290b3fa64bd571fb WatchSource:0}: Error finding container a71555022b802ad458fa97cc5631d0abac8bfe5e8b7ccd01290b3fa64bd571fb: Status 404 returned error can't find the container with id a71555022b802ad458fa97cc5631d0abac8bfe5e8b7ccd01290b3fa64bd571fb Oct 31 13:28:49.280030 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.279995 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:28:49.280134 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.280073 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:28:49.280134 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.280118 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:28:49.319611 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.319585 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxn2\" (UniqueName: \"kubernetes.io/projected/fa049032-484c-43c5-9440-7928760a0967-kube-api-access-jcxn2\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.319734 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.319648 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa049032-484c-43c5-9440-7928760a0967-openshift-service-ca\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.319734 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.319676 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa049032-484c-43c5-9440-7928760a0967-maas-api-tls\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.319844 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.319745 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa049032-484c-43c5-9440-7928760a0967-metrics-certs\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.320346 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.320324 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa049032-484c-43c5-9440-7928760a0967-openshift-service-ca\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.322118 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.322099 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa049032-484c-43c5-9440-7928760a0967-metrics-certs\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.322206 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.322124 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa049032-484c-43c5-9440-7928760a0967-maas-api-tls\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.334077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.334051 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxn2\" (UniqueName: \"kubernetes.io/projected/fa049032-484c-43c5-9440-7928760a0967-kube-api-access-jcxn2\") pod \"maas-api-7df6dd5c76-crvv7\" (UID: \"fa049032-484c-43c5-9440-7928760a0967\") " pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.422450 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.422416 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:49.699862 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.699838 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7"] Oct 31 13:28:49.702239 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:28:49.702208 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa049032_484c_43c5_9440_7928760a0967.slice/crio-4c764e9ef251774a29486c0e076504143204e17cbb9f147386c7d8f2f2dcdd59 WatchSource:0}: Error finding container 4c764e9ef251774a29486c0e076504143204e17cbb9f147386c7d8f2f2dcdd59: Status 404 returned error can't find the container with id 4c764e9ef251774a29486c0e076504143204e17cbb9f147386c7d8f2f2dcdd59 Oct 31 13:28:49.703726 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:49.703709 4542 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:28:50.055977 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:50.055936 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" event={"ID":"47ab6adf-84d6-4b87-975c-31ecabe6f630","Type":"ContainerStarted","Data":"9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f"} Oct 31 13:28:50.056168 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:50.056043 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" event={"ID":"47ab6adf-84d6-4b87-975c-31ecabe6f630","Type":"ContainerStarted","Data":"a71555022b802ad458fa97cc5631d0abac8bfe5e8b7ccd01290b3fa64bd571fb"} Oct 31 13:28:50.057168 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:50.057145 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" event={"ID":"fa049032-484c-43c5-9440-7928760a0967","Type":"ContainerStarted","Data":"4c764e9ef251774a29486c0e076504143204e17cbb9f147386c7d8f2f2dcdd59"} Oct 31 13:28:50.084077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:50.084032 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" podStartSLOduration=2.084016782 podStartE2EDuration="2.084016782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:50.083338734 +0000 UTC m=+1746.928505153" watchObservedRunningTime="2026-10-31 13:28:50.084016782 +0000 UTC m=+1746.929183255" Oct 31 13:28:50.990323 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:50.990295 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:50.995497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:50.995451 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:51.063860 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:51.063824 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" event={"ID":"fa049032-484c-43c5-9440-7928760a0967","Type":"ContainerStarted","Data":"f465e9d50ab223bcdec852446250c00f35221d6c53d8717bb248a3fb7db75bda"} Oct 31 13:28:51.064051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:51.063903 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:51.064051 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:51.063940 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:51.064873 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:51.064854 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:28:51.086455 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:51.086412 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" podStartSLOduration=1.631434152 podStartE2EDuration="2.086398616s" totalImagesPullingTime="454.964464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:51.08532451 +0000 UTC m=+1747.930490929" watchObservedRunningTime="2026-10-31 13:28:51.086398616 +0000 UTC m=+1747.931565034" Oct 31 13:28:53.028951 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.028917 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj"] Oct 31 13:28:53.032469 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.032453 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:28:53.043195 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.043166 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj"] Oct 31 13:28:53.043454 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.043432 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-351178df\"" Oct 31 13:28:53.052845 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.052823 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj\" (UID: \"5431e79c-f350-4cf1-a87c-3d876b4eb1e4\") " pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:28:53.052966 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.052904 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7pwh\" (UniqueName: \"kubernetes.io/projected/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-kube-api-access-q7pwh\") pod \"payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj\" (UID: \"5431e79c-f350-4cf1-a87c-3d876b4eb1e4\") " pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:28:53.154141 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.154108 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj\" (UID: \"5431e79c-f350-4cf1-a87c-3d876b4eb1e4\") " pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:28:53.154343 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.154218 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7pwh\" (UniqueName: \"kubernetes.io/projected/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-kube-api-access-q7pwh\") pod \"payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj\" (UID: \"5431e79c-f350-4cf1-a87c-3d876b4eb1e4\") " pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:28:53.155529 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.155452 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj\" (UID: \"5431e79c-f350-4cf1-a87c-3d876b4eb1e4\") " pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:28:53.168352 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.168320 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7pwh\" (UniqueName: \"kubernetes.io/projected/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-kube-api-access-q7pwh\") pod \"payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj\" (UID: \"5431e79c-f350-4cf1-a87c-3d876b4eb1e4\") " pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:28:53.346671 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.346636 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:28:53.640607 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:53.640582 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj"] Oct 31 13:28:53.643215 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:28:53.643183 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5431e79c_f350_4cf1_a87c_3d876b4eb1e4.slice/crio-24c8f655e2ba97af4914aa84c71c9ede306b2047a49add672b1822409afbd139 WatchSource:0}: Error finding container 24c8f655e2ba97af4914aa84c71c9ede306b2047a49add672b1822409afbd139: Status 404 returned error can't find the container with id 24c8f655e2ba97af4914aa84c71c9ede306b2047a49add672b1822409afbd139 Oct 31 13:28:53.844858 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:28:53.844784 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:28:54.077972 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:54.077938 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" event={"ID":"5431e79c-f350-4cf1-a87c-3d876b4eb1e4","Type":"ContainerStarted","Data":"715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9"} Oct 31 13:28:54.078381 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:54.078016 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" event={"ID":"5431e79c-f350-4cf1-a87c-3d876b4eb1e4","Type":"ContainerStarted","Data":"24c8f655e2ba97af4914aa84c71c9ede306b2047a49add672b1822409afbd139"} Oct 31 13:28:54.078381 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:54.078072 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:28:54.098578 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:54.098532 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" podStartSLOduration=1.098519067 podStartE2EDuration="1.098519067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:54.09701154 +0000 UTC m=+1750.942177959" watchObservedRunningTime="2026-10-31 13:28:54.098519067 +0000 UTC m=+1750.943685484" Oct 31 13:28:55.403191 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:55.403156 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:28:57.074457 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:57.074382 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7df6dd5c76-crvv7" Oct 31 13:28:57.128878 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:57.128836 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq"] Oct 31 13:28:57.129117 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:57.129082 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" podUID="c15373be-9ca7-4874-8cf6-f5187d341e89" containerName="maas-api" containerID="cri-o://0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2" gracePeriod=30 Oct 31 13:28:58.419645 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.419620 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:28:58.506429 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.506344 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-metrics-certs\") pod \"c15373be-9ca7-4874-8cf6-f5187d341e89\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " Oct 31 13:28:58.506429 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.506378 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-maas-api-tls\") pod \"c15373be-9ca7-4874-8cf6-f5187d341e89\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " Oct 31 13:28:58.506634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.506432 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15373be-9ca7-4874-8cf6-f5187d341e89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15373be-9ca7-4874-8cf6-f5187d341e89-openshift-service-ca\") pod \"c15373be-9ca7-4874-8cf6-f5187d341e89\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " Oct 31 13:28:58.506669 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.506632 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15373be-9ca7-4874-8cf6-f5187d341e89-kube-api-access-trc8h\" (UniqueName: \"kubernetes.io/projected/c15373be-9ca7-4874-8cf6-f5187d341e89-kube-api-access-trc8h\") pod \"c15373be-9ca7-4874-8cf6-f5187d341e89\" (UID: \"c15373be-9ca7-4874-8cf6-f5187d341e89\") " Oct 31 13:28:58.506911 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.506879 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15373be-9ca7-4874-8cf6-f5187d341e89-openshift-service-ca" pod "c15373be-9ca7-4874-8cf6-f5187d341e89" (UID: "c15373be-9ca7-4874-8cf6-f5187d341e89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:58.508615 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.508590 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-maas-api-tls" pod "c15373be-9ca7-4874-8cf6-f5187d341e89" (UID: "c15373be-9ca7-4874-8cf6-f5187d341e89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:28:58.508615 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.508606 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-metrics-certs" pod "c15373be-9ca7-4874-8cf6-f5187d341e89" (UID: "c15373be-9ca7-4874-8cf6-f5187d341e89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:28:58.508767 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.508701 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15373be-9ca7-4874-8cf6-f5187d341e89-kube-api-access-trc8h" pod "c15373be-9ca7-4874-8cf6-f5187d341e89" (UID: "c15373be-9ca7-4874-8cf6-f5187d341e89"). InnerVolumeSpecName "kube-api-access-trc8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:58.607553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.607521 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trc8h\" (UniqueName: \"kubernetes.io/projected/c15373be-9ca7-4874-8cf6-f5187d341e89-kube-api-access-trc8h\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.607553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.607551 4542 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-metrics-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.607553 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.607561 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15373be-9ca7-4874-8cf6-f5187d341e89-maas-api-tls\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:58.607781 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:58.607570 4542 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15373be-9ca7-4874-8cf6-f5187d341e89-openshift-service-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:59.102654 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:59.102623 4542 generic.go:356] "Generic (PLEG): container finished" podID="c15373be-9ca7-4874-8cf6-f5187d341e89" containerID="0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2" exitCode=0 Oct 31 13:28:59.102846 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:59.102670 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" Oct 31 13:28:59.102846 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:59.102714 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" event={"ID":"c15373be-9ca7-4874-8cf6-f5187d341e89","Type":"ContainerDied","Data":"0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2"} Oct 31 13:28:59.102846 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:59.102771 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq" event={"ID":"c15373be-9ca7-4874-8cf6-f5187d341e89","Type":"ContainerDied","Data":"d9e71a8c63ce2f6cac89f6a363305e84ecde5f3a0be63b21222710870353a0bd"} Oct 31 13:28:59.102846 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:59.102801 4542 scope.go:122] "RemoveContainer" containerID="0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2" Oct 31 13:28:59.112369 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:59.112351 4542 scope.go:122] "RemoveContainer" containerID="0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2" Oct 31 13:28:59.112690 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:28:59.112670 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2\": container with ID starting with 0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2 not found: ID does not exist" containerID="0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2" Oct 31 13:28:59.112742 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:59.112701 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2"} err="failed to get container status \"0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2\": rpc error: code = NotFound desc = could not find container \"0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2\": container with ID starting with 0d70a72bcd1f1e4b10665d4d80fc267b7a1b627b827f6fdba0a21bab5d85a5d2 not found: ID does not exist" Oct 31 13:28:59.125691 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:59.125667 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq"] Oct 31 13:28:59.130082 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:59.130055 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-54d9cc59bd-dlrpq"] Oct 31 13:28:59.290125 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:28:59.290090 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15373be-9ca7-4874-8cf6-f5187d341e89" path="/var/lib/kubelet/pods/c15373be-9ca7-4874-8cf6-f5187d341e89/volumes" Oct 31 13:29:05.086026 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:05.085989 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:29:05.135091 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:05.135001 4542 generic.go:356] "Generic (PLEG): container finished" podID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerID="9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f" exitCode=0 Oct 31 13:29:05.135091 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:05.135043 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" event={"ID":"47ab6adf-84d6-4b87-975c-31ecabe6f630","Type":"ContainerDied","Data":"9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f"} Oct 31 13:29:05.135434 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:05.135417 4542 scope.go:122] "RemoveContainer" containerID="9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f" Oct 31 13:29:05.136689 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:05.136659 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:29:05.136770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:05.136727 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:29:05.136770 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:05.136763 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:29:06.064643 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:06.064597 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:29:06.167957 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:06.167919 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" event={"ID":"47ab6adf-84d6-4b87-975c-31ecabe6f630","Type":"ContainerStarted","Data":"fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab"} Oct 31 13:29:06.990798 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:06.990761 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:29:06.991993 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:06.991968 4542 patch_prober.go:29] interesting pod/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 13:29:06.992054 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:06.992025 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 13:29:07.991201 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:07.991163 4542 patch_prober.go:29] interesting pod/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 13:29:07.991727 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:07.991223 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 13:29:08.990873 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:08.990839 4542 patch_prober.go:29] interesting pod/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 13:29:08.991065 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:08.990894 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 13:29:09.991027 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:09.990990 4542 patch_prober.go:29] interesting pod/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 13:29:09.991544 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:09.991048 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 13:29:10.991247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:10.991214 4542 patch_prober.go:29] interesting pod/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 13:29:10.991785 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:10.991273 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 13:29:11.994311 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:11.994276 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:29:11.994802 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:11.994669 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:29:11.995454 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:11.995433 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:29:17.639318 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:17.639276 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj"] Oct 31 13:29:19.326122 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:19.326094 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:29:22.640628 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:22.640570 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" podUID="5431e79c-f350-4cf1-a87c-3d876b4eb1e4" containerName="payload-processing" containerID="cri-o://715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9" gracePeriod=25 Oct 31 13:29:22.896779 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:22.896715 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:29:22.997939 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:22.997905 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-kube-api-access-q7pwh\" (UniqueName: \"kubernetes.io/projected/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-kube-api-access-q7pwh\") pod \"5431e79c-f350-4cf1-a87c-3d876b4eb1e4\" (UID: \"5431e79c-f350-4cf1-a87c-3d876b4eb1e4\") " Oct 31 13:29:22.998117 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:22.997965 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-plugins-config-volume\") pod \"5431e79c-f350-4cf1-a87c-3d876b4eb1e4\" (UID: \"5431e79c-f350-4cf1-a87c-3d876b4eb1e4\") " Oct 31 13:29:22.998374 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:22.998348 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-plugins-config-volume" pod "5431e79c-f350-4cf1-a87c-3d876b4eb1e4" (UID: "5431e79c-f350-4cf1-a87c-3d876b4eb1e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:23.000029 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.000009 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-kube-api-access-q7pwh" pod "5431e79c-f350-4cf1-a87c-3d876b4eb1e4" (UID: "5431e79c-f350-4cf1-a87c-3d876b4eb1e4"). InnerVolumeSpecName "kube-api-access-q7pwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:23.099654 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.099627 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7pwh\" (UniqueName: \"kubernetes.io/projected/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-kube-api-access-q7pwh\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:23.099654 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.099650 4542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5431e79c-f350-4cf1-a87c-3d876b4eb1e4-plugins-config-volume\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:23.248390 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.248299 4542 generic.go:356] "Generic (PLEG): container finished" podID="5431e79c-f350-4cf1-a87c-3d876b4eb1e4" containerID="715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9" exitCode=0 Oct 31 13:29:23.248390 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.248346 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" Oct 31 13:29:23.248390 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.248375 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" event={"ID":"5431e79c-f350-4cf1-a87c-3d876b4eb1e4","Type":"ContainerDied","Data":"715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9"} Oct 31 13:29:23.248703 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.248450 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj" event={"ID":"5431e79c-f350-4cf1-a87c-3d876b4eb1e4","Type":"ContainerDied","Data":"24c8f655e2ba97af4914aa84c71c9ede306b2047a49add672b1822409afbd139"} Oct 31 13:29:23.248703 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.248501 4542 scope.go:122] "RemoveContainer" containerID="715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9" Oct 31 13:29:23.257128 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.257110 4542 scope.go:122] "RemoveContainer" containerID="715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9" Oct 31 13:29:23.257412 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:29:23.257394 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9\": container with ID starting with 715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9 not found: ID does not exist" containerID="715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9" Oct 31 13:29:23.257459 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.257420 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9"} err="failed to get container status \"715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9\": rpc error: code = NotFound desc = could not find container \"715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9\": container with ID starting with 715adc81aa657ff1e4ee5fe25ce77f7933992ef0f06006a08a884cb3c25d70e9 not found: ID does not exist" Oct 31 13:29:23.271975 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.271950 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj"] Oct 31 13:29:23.276808 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.276788 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-351178df-6f6cf84b5-7q5tj"] Oct 31 13:29:23.290017 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.289993 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5431e79c-f350-4cf1-a87c-3d876b4eb1e4" path="/var/lib/kubelet/pods/5431e79c-f350-4cf1-a87c-3d876b4eb1e4/volumes" Oct 31 13:29:23.831215 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:29:23.831184 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:29:23.904629 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.904582 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9"] Oct 31 13:29:23.904920 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:23.904885 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" containerID="cri-o://fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab" gracePeriod=30 Oct 31 13:29:25.662939 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.662908 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8"] Oct 31 13:29:25.663328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.663277 4542 cpu_manager.go:414] "containerMap: removing container" podUID="5431e79c-f350-4cf1-a87c-3d876b4eb1e4" containerName="payload-processing" Oct 31 13:29:25.663328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.663288 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5431e79c-f350-4cf1-a87c-3d876b4eb1e4" containerName="payload-processing" Oct 31 13:29:25.663328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.663301 4542 cpu_manager.go:414] "containerMap: removing container" podUID="c15373be-9ca7-4874-8cf6-f5187d341e89" containerName="maas-api" Oct 31 13:29:25.663328 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.663307 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15373be-9ca7-4874-8cf6-f5187d341e89" containerName="maas-api" Oct 31 13:29:25.663458 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.663389 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="5431e79c-f350-4cf1-a87c-3d876b4eb1e4" containerName="payload-processing" Oct 31 13:29:25.663458 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.663400 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15373be-9ca7-4874-8cf6-f5187d341e89" containerName="maas-api" Oct 31 13:29:25.667941 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.667918 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.675901 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.675881 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-9c81e5-openshift-default-dockercfg-qfcmq\"" Oct 31 13:29:25.683459 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.683435 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8"] Oct 31 13:29:25.724513 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.724448 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-socket\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.725163 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.724938 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-envoy\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.725163 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.725008 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-certs\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.725163 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.725041 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7470834a-3540-460d-ac9b-5133bd2c7132-istio-podinfo\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.725163 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.725092 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2lz\" (UniqueName: \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-kube-api-access-hc2lz\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.725163 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.725125 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7470834a-3540-460d-ac9b-5133bd2c7132-istiod-ca-cert\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.725498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.725189 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-credential-socket\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.725498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.725245 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-istio-token\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.725498 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.725296 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-data\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.826787 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.826746 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-data\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.826999 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.826804 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-socket\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.826999 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.826854 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-envoy\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.826999 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.826889 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-certs\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.826999 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.826923 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7470834a-3540-460d-ac9b-5133bd2c7132-istio-podinfo\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.826999 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.826969 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2lz\" (UniqueName: \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-kube-api-access-hc2lz\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.827282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.827004 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7470834a-3540-460d-ac9b-5133bd2c7132-istiod-ca-cert\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.827282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.827063 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-credential-socket\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.827282 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.827114 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-istio-token\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.827436 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.827316 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-data\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.828278 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.828250 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7470834a-3540-460d-ac9b-5133bd2c7132-istiod-ca-cert\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.828447 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.828259 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-socket\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.828447 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.828314 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-credential-socket\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.828646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.828501 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-certs\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.830551 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.830522 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7470834a-3540-460d-ac9b-5133bd2c7132-istio-podinfo\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.833087 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.833045 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-envoy\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.842865 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.842837 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2lz\" (UniqueName: \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-kube-api-access-hc2lz\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.848400 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.848368 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-istio-token\") pod \"e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:25.981526 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:25.981412 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:26.263776 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:26.263750 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8"] Oct 31 13:29:26.265421 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:29:26.265393 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7470834a_3540_460d_ac9b_5133bd2c7132.slice/crio-c7a4c74be12e5263bd0854763b240d1911e8723f2108637c06a2ddb7eaa17262 WatchSource:0}: Error finding container c7a4c74be12e5263bd0854763b240d1911e8723f2108637c06a2ddb7eaa17262: Status 404 returned error can't find the container with id c7a4c74be12e5263bd0854763b240d1911e8723f2108637c06a2ddb7eaa17262 Oct 31 13:29:26.267720 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:26.267686 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:29:26.267804 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:26.267753 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:29:26.267846 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:26.267803 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:29:26.996518 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:26.996463 4542 patch_prober.go:29] interesting pod/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:29:26.996882 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:26.996547 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:29:27.269255 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:27.269172 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" event={"ID":"7470834a-3540-460d-ac9b-5133bd2c7132","Type":"ContainerStarted","Data":"a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7"} Oct 31 13:29:27.269391 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:27.269272 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" event={"ID":"7470834a-3540-460d-ac9b-5133bd2c7132","Type":"ContainerStarted","Data":"c7a4c74be12e5263bd0854763b240d1911e8723f2108637c06a2ddb7eaa17262"} Oct 31 13:29:27.295821 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:27.295775 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" podStartSLOduration=2.295762373 podStartE2EDuration="2.295762373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:27.29442902 +0000 UTC m=+1784.139595441" watchObservedRunningTime="2026-10-31 13:29:27.295762373 +0000 UTC m=+1784.140928791" Oct 31 13:29:27.981826 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:27.981792 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:27.986980 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:27.986953 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:28.273542 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:28.273433 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:28.274644 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:28.274627 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:29.149644 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.149615 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:29:29.264014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.263945 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-kube-api-access-js7z2\" (UniqueName: \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-kube-api-access-js7z2\") pod \"47ab6adf-84d6-4b87-975c-31ecabe6f630\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " Oct 31 13:29:29.264014 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.263978 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-podinfo\") pod \"47ab6adf-84d6-4b87-975c-31ecabe6f630\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " Oct 31 13:29:29.264214 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264034 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-envoy\") pod \"47ab6adf-84d6-4b87-975c-31ecabe6f630\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " Oct 31 13:29:29.264214 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264105 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-socket\") pod \"47ab6adf-84d6-4b87-975c-31ecabe6f630\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " Oct 31 13:29:29.264214 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264135 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ab6adf-84d6-4b87-975c-31ecabe6f630-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ab6adf-84d6-4b87-975c-31ecabe6f630-istiod-ca-cert\") pod \"47ab6adf-84d6-4b87-975c-31ecabe6f630\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " Oct 31 13:29:29.264378 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264212 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-certs\") pod \"47ab6adf-84d6-4b87-975c-31ecabe6f630\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " Oct 31 13:29:29.264378 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264247 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-data\") pod \"47ab6adf-84d6-4b87-975c-31ecabe6f630\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " Oct 31 13:29:29.264378 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264285 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-credential-socket\") pod \"47ab6adf-84d6-4b87-975c-31ecabe6f630\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " Oct 31 13:29:29.264378 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264328 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-token\" (UniqueName: \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-token\") pod \"47ab6adf-84d6-4b87-975c-31ecabe6f630\" (UID: \"47ab6adf-84d6-4b87-975c-31ecabe6f630\") " Oct 31 13:29:29.264625 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264443 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-socket" pod "47ab6adf-84d6-4b87-975c-31ecabe6f630" (UID: "47ab6adf-84d6-4b87-975c-31ecabe6f630"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:29.264680 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264637 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ab6adf-84d6-4b87-975c-31ecabe6f630-istiod-ca-cert" pod "47ab6adf-84d6-4b87-975c-31ecabe6f630" (UID: "47ab6adf-84d6-4b87-975c-31ecabe6f630"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:29.264740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264686 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-credential-socket" pod "47ab6adf-84d6-4b87-975c-31ecabe6f630" (UID: "47ab6adf-84d6-4b87-975c-31ecabe6f630"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:29.264740 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264706 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-data" pod "47ab6adf-84d6-4b87-975c-31ecabe6f630" (UID: "47ab6adf-84d6-4b87-975c-31ecabe6f630"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:29.264838 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264774 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-certs" pod "47ab6adf-84d6-4b87-975c-31ecabe6f630" (UID: "47ab6adf-84d6-4b87-975c-31ecabe6f630"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:29.264896 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264875 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:29.264938 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264898 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ab6adf-84d6-4b87-975c-31ecabe6f630-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:29.264938 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264913 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:29.264938 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264927 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:29.265033 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.264940 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:29.266247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.266218 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-envoy" pod "47ab6adf-84d6-4b87-975c-31ecabe6f630" (UID: "47ab6adf-84d6-4b87-975c-31ecabe6f630"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:29.266366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.266284 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-kube-api-access-js7z2" pod "47ab6adf-84d6-4b87-975c-31ecabe6f630" (UID: "47ab6adf-84d6-4b87-975c-31ecabe6f630"). InnerVolumeSpecName "kube-api-access-js7z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:29.266544 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.266528 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-podinfo" pod "47ab6adf-84d6-4b87-975c-31ecabe6f630" (UID: "47ab6adf-84d6-4b87-975c-31ecabe6f630"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:29:29.266726 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.266710 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-token" pod "47ab6adf-84d6-4b87-975c-31ecabe6f630" (UID: "47ab6adf-84d6-4b87-975c-31ecabe6f630"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:29.279543 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.279521 4542 generic.go:356] "Generic (PLEG): container finished" podID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerID="fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab" exitCode=0 Oct 31 13:29:29.279935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.279577 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" Oct 31 13:29:29.279935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.279601 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" event={"ID":"47ab6adf-84d6-4b87-975c-31ecabe6f630","Type":"ContainerDied","Data":"fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab"} Oct 31 13:29:29.279935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.279663 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9" event={"ID":"47ab6adf-84d6-4b87-975c-31ecabe6f630","Type":"ContainerDied","Data":"a71555022b802ad458fa97cc5631d0abac8bfe5e8b7ccd01290b3fa64bd571fb"} Oct 31 13:29:29.279935 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.279693 4542 scope.go:122] "RemoveContainer" containerID="fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab" Oct 31 13:29:29.289179 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.289161 4542 scope.go:122] "RemoveContainer" containerID="9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f" Oct 31 13:29:29.296816 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.296798 4542 scope.go:122] "RemoveContainer" containerID="fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab" Oct 31 13:29:29.297075 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:29:29.297057 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab\": container with ID starting with fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab not found: ID does not exist" containerID="fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab" Oct 31 13:29:29.297133 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.297083 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab"} err="failed to get container status \"fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab\": rpc error: code = NotFound desc = could not find container \"fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab\": container with ID starting with fb0aea87233ec25066dc1075062ec5401cc690721deb8b298cd3b5fbd8aaf4ab not found: ID does not exist" Oct 31 13:29:29.297133 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.297101 4542 scope.go:122] "RemoveContainer" containerID="9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f" Oct 31 13:29:29.297339 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:29:29.297320 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f\": container with ID starting with 9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f not found: ID does not exist" containerID="9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f" Oct 31 13:29:29.297395 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.297348 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f"} err="failed to get container status \"9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f\": rpc error: code = NotFound desc = could not find container \"9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f\": container with ID starting with 9d7777a4b2663ab40297b4ca4db7807921a5e402899ec32878ffd35b105c7a9f not found: ID does not exist" Oct 31 13:29:29.303789 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.303769 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9"] Oct 31 13:29:29.306032 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.306013 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-351178df-openshift-default-5cd8548d8c-npfd9"] Oct 31 13:29:29.366527 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.366466 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js7z2\" (UniqueName: \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-kube-api-access-js7z2\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:29.366635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.366533 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:29.366635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.366551 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:29.366635 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:29.366566 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ab6adf-84d6-4b87-975c-31ecabe6f630-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:30.908595 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.908558 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4"] Oct 31 13:29:30.909066 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.909045 4542 cpu_manager.go:414] "containerMap: removing container" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" Oct 31 13:29:30.909066 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.909060 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" Oct 31 13:29:30.909188 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.909074 4542 cpu_manager.go:414] "containerMap: removing container" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" Oct 31 13:29:30.909188 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.909079 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" Oct 31 13:29:30.909188 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.909161 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" Oct 31 13:29:30.909188 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.909172 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" containerName="istio-proxy" Oct 31 13:29:30.914111 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.914087 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:29:30.927116 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.927094 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-9c81e5\"" Oct 31 13:29:30.930162 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.930125 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-9c81e5-dockercfg-cj5w6\"" Oct 31 13:29:30.930869 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.930842 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4"] Oct 31 13:29:30.944064 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.944007 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg"] Oct 31 13:29:30.955874 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.953881 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:29:30.965033 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.964986 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg"] Oct 31 13:29:30.982154 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.982127 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvxv\" (UniqueName: \"kubernetes.io/projected/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-kube-api-access-whvxv\") pod \"payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4\" (UID: \"f0e4e4ab-32eb-4a22-97d4-641d98f034a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:29:30.982294 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:30.982252 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4\" (UID: \"f0e4e4ab-32eb-4a22-97d4-641d98f034a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:29:31.083619 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.083561 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4\" (UID: \"f0e4e4ab-32eb-4a22-97d4-641d98f034a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:29:31.083805 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.083672 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6880566-9beb-4933-869a-4bc3ce78b016-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg\" (UID: \"c6880566-9beb-4933-869a-4bc3ce78b016\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:29:31.083805 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.083760 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92zph\" (UniqueName: \"kubernetes.io/projected/c6880566-9beb-4933-869a-4bc3ce78b016-kube-api-access-92zph\") pod \"payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg\" (UID: \"c6880566-9beb-4933-869a-4bc3ce78b016\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:29:31.083805 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.083786 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvxv\" (UniqueName: \"kubernetes.io/projected/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-kube-api-access-whvxv\") pod \"payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4\" (UID: \"f0e4e4ab-32eb-4a22-97d4-641d98f034a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:29:31.084423 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.084402 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4\" (UID: \"f0e4e4ab-32eb-4a22-97d4-641d98f034a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:29:31.097900 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.097867 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvxv\" (UniqueName: \"kubernetes.io/projected/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-kube-api-access-whvxv\") pod \"payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4\" (UID: \"f0e4e4ab-32eb-4a22-97d4-641d98f034a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:29:31.184497 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.184418 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6880566-9beb-4933-869a-4bc3ce78b016-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg\" (UID: \"c6880566-9beb-4933-869a-4bc3ce78b016\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:29:31.184632 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.184497 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92zph\" (UniqueName: \"kubernetes.io/projected/c6880566-9beb-4933-869a-4bc3ce78b016-kube-api-access-92zph\") pod \"payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg\" (UID: \"c6880566-9beb-4933-869a-4bc3ce78b016\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:29:31.185077 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.185058 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6880566-9beb-4933-869a-4bc3ce78b016-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg\" (UID: \"c6880566-9beb-4933-869a-4bc3ce78b016\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:29:31.197083 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.197054 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92zph\" (UniqueName: \"kubernetes.io/projected/c6880566-9beb-4933-869a-4bc3ce78b016-kube-api-access-92zph\") pod \"payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg\" (UID: \"c6880566-9beb-4933-869a-4bc3ce78b016\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:29:31.233646 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.233612 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:29:31.270802 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.270768 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:29:31.298962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.298926 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ab6adf-84d6-4b87-975c-31ecabe6f630" path="/var/lib/kubelet/pods/47ab6adf-84d6-4b87-975c-31ecabe6f630/volumes" Oct 31 13:29:31.516906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.516829 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4"] Oct 31 13:29:31.523631 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:29:31.523600 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e4e4ab_32eb_4a22_97d4_641d98f034a2.slice/crio-30f994ee1ae7427f9aa2a7e75e74807a945d33ccc40d06eceb4f2e39b91178db WatchSource:0}: Error finding container 30f994ee1ae7427f9aa2a7e75e74807a945d33ccc40d06eceb4f2e39b91178db: Status 404 returned error can't find the container with id 30f994ee1ae7427f9aa2a7e75e74807a945d33ccc40d06eceb4f2e39b91178db Oct 31 13:29:31.559868 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:31.559749 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg"] Oct 31 13:29:31.561881 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:29:31.561846 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6880566_9beb_4933_869a_4bc3ce78b016.slice/crio-6cdfbbfbe7882abd73b81baff81703bbc3ae7319dbba847c2927f10d4fc497e7 WatchSource:0}: Error finding container 6cdfbbfbe7882abd73b81baff81703bbc3ae7319dbba847c2927f10d4fc497e7: Status 404 returned error can't find the container with id 6cdfbbfbe7882abd73b81baff81703bbc3ae7319dbba847c2927f10d4fc497e7 Oct 31 13:29:32.302244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:32.302206 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" event={"ID":"c6880566-9beb-4933-869a-4bc3ce78b016","Type":"ContainerStarted","Data":"57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6"} Oct 31 13:29:32.302729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:32.302270 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" event={"ID":"c6880566-9beb-4933-869a-4bc3ce78b016","Type":"ContainerStarted","Data":"6cdfbbfbe7882abd73b81baff81703bbc3ae7319dbba847c2927f10d4fc497e7"} Oct 31 13:29:32.302729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:32.302302 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:29:32.303736 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:32.303709 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" event={"ID":"f0e4e4ab-32eb-4a22-97d4-641d98f034a2","Type":"ContainerStarted","Data":"d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3"} Oct 31 13:29:32.303905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:32.303752 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" event={"ID":"f0e4e4ab-32eb-4a22-97d4-641d98f034a2","Type":"ContainerStarted","Data":"30f994ee1ae7427f9aa2a7e75e74807a945d33ccc40d06eceb4f2e39b91178db"} Oct 31 13:29:32.303905 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:32.303782 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:29:32.322560 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:32.322510 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" podStartSLOduration=2.322468378 podStartE2EDuration="2.322468378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:32.321810136 +0000 UTC m=+1789.166976557" watchObservedRunningTime="2026-10-31 13:29:32.322468378 +0000 UTC m=+1789.167634797" Oct 31 13:29:32.342184 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:32.342136 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" podStartSLOduration=2.3421253220000002 podStartE2EDuration="2.342125322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:32.340630655 +0000 UTC m=+1789.185797075" watchObservedRunningTime="2026-10-31 13:29:32.342125322 +0000 UTC m=+1789.187291736" Oct 31 13:29:33.130517 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:33.130469 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:29:42.418777 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:29:42.418743 4542 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7470834a_3540_460d_ac9b_5133bd2c7132.slice/crio-a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7470834a_3540_460d_ac9b_5133bd2c7132.slice/crio-conmon-a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:29:43.274906 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.274867 4542 patch_prober.go:29] interesting pod/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 13:29:43.275207 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.274923 4542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 13:29:43.311581 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.311552 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:29:43.311767 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.311629 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:29:43.360043 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.360013 4542 generic.go:356] "Generic (PLEG): container finished" podID="7470834a-3540-460d-ac9b-5133bd2c7132" containerID="a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7" exitCode=0 Oct 31 13:29:43.360244 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.360125 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" event={"ID":"7470834a-3540-460d-ac9b-5133bd2c7132","Type":"ContainerDied","Data":"a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7"} Oct 31 13:29:43.360567 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.360545 4542 scope.go:122] "RemoveContainer" containerID="a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7" Oct 31 13:29:43.361884 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.361847 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:29:43.361988 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.361932 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:29:43.361988 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.361975 4542 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236056Ki","pods":"250"} Oct 31 13:29:43.725286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.725258 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:29:43.730359 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:43.730332 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:29:44.369567 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:44.369531 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" event={"ID":"7470834a-3540-460d-ac9b-5133bd2c7132","Type":"ContainerStarted","Data":"18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518"} Oct 31 13:29:44.982550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:44.982520 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:44.982956 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:44.982776 4542 patch_prober.go:29] interesting pod/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 13:29:44.982956 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:44.982818 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 13:29:45.982783 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:45.982741 4542 patch_prober.go:29] interesting pod/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 13:29:45.983198 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:45.982802 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 13:29:46.982301 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:46.982262 4542 patch_prober.go:29] interesting pod/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 13:29:46.982538 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:46.982319 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 13:29:47.982751 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:47.982719 4542 patch_prober.go:29] interesting pod/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 13:29:47.983145 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:47.982771 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 13:29:48.982719 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:48.982682 4542 patch_prober.go:29] interesting pod/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 13:29:48.982907 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:48.982738 4542 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 13:29:49.986757 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:49.986725 4542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:49.987179 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:49.986978 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:49.987717 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:29:49.987699 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:29:53.832073 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:29:53.832037 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:30:00.147881 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.147845 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj"] Oct 31 13:30:00.152525 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.152507 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" Oct 31 13:30:00.161525 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.161459 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj"] Oct 31 13:30:00.201844 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.201818 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305d94b0-451d-446a-acdf-51f5acb18a15-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890890-wcbhj\" (UID: \"305d94b0-451d-446a-acdf-51f5acb18a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" Oct 31 13:30:00.201965 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.201857 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k89sh\" (UniqueName: \"kubernetes.io/projected/305d94b0-451d-446a-acdf-51f5acb18a15-kube-api-access-k89sh\") pod \"maas-api-key-cleanup-29890890-wcbhj\" (UID: \"305d94b0-451d-446a-acdf-51f5acb18a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" Oct 31 13:30:00.256668 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.256645 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z5j9\"" Oct 31 13:30:00.302286 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.302264 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k89sh\" (UniqueName: \"kubernetes.io/projected/305d94b0-451d-446a-acdf-51f5acb18a15-kube-api-access-k89sh\") pod \"maas-api-key-cleanup-29890890-wcbhj\" (UID: \"305d94b0-451d-446a-acdf-51f5acb18a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" Oct 31 13:30:00.302431 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.302411 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305d94b0-451d-446a-acdf-51f5acb18a15-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890890-wcbhj\" (UID: \"305d94b0-451d-446a-acdf-51f5acb18a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" Oct 31 13:30:00.303089 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.303068 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305d94b0-451d-446a-acdf-51f5acb18a15-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890890-wcbhj\" (UID: \"305d94b0-451d-446a-acdf-51f5acb18a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" Oct 31 13:30:00.316080 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.316052 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k89sh\" (UniqueName: \"kubernetes.io/projected/305d94b0-451d-446a-acdf-51f5acb18a15-kube-api-access-k89sh\") pod \"maas-api-key-cleanup-29890890-wcbhj\" (UID: \"305d94b0-451d-446a-acdf-51f5acb18a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" Oct 31 13:30:00.468994 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.468927 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" Oct 31 13:30:00.741372 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:00.741350 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj"] Oct 31 13:30:00.743905 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:30:00.743878 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305d94b0_451d_446a_acdf_51f5acb18a15.slice/crio-71cef74444c0aaca02060a3bbf868f92b92b9535e94f569eca4c5962486feeb2 WatchSource:0}: Error finding container 71cef74444c0aaca02060a3bbf868f92b92b9535e94f569eca4c5962486feeb2: Status 404 returned error can't find the container with id 71cef74444c0aaca02060a3bbf868f92b92b9535e94f569eca4c5962486feeb2 Oct 31 13:30:01.454895 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:01.454856 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" event={"ID":"305d94b0-451d-446a-acdf-51f5acb18a15","Type":"ContainerStarted","Data":"03db179f834a47bdea97bf50ae295bd403f6abb3dace5f5f5558e04714fc0ff5"} Oct 31 13:30:01.455382 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:01.454913 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" event={"ID":"305d94b0-451d-446a-acdf-51f5acb18a15","Type":"ContainerStarted","Data":"71cef74444c0aaca02060a3bbf868f92b92b9535e94f569eca4c5962486feeb2"} Oct 31 13:30:01.474893 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:01.474848 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" podStartSLOduration=1.474833964 podStartE2EDuration="1.474833964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:01.473503365 +0000 UTC m=+1818.318669779" watchObservedRunningTime="2026-10-31 13:30:01.474833964 +0000 UTC m=+1818.320000381" Oct 31 13:30:02.461723 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:02.461685 4542 generic.go:356] "Generic (PLEG): container finished" podID="305d94b0-451d-446a-acdf-51f5acb18a15" containerID="03db179f834a47bdea97bf50ae295bd403f6abb3dace5f5f5558e04714fc0ff5" exitCode=60 Oct 31 13:30:02.462121 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:02.461736 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" event={"ID":"305d94b0-451d-446a-acdf-51f5acb18a15","Type":"ContainerDied","Data":"03db179f834a47bdea97bf50ae295bd403f6abb3dace5f5f5558e04714fc0ff5"} Oct 31 13:30:02.462121 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:02.461990 4542 scope.go:122] "RemoveContainer" containerID="03db179f834a47bdea97bf50ae295bd403f6abb3dace5f5f5558e04714fc0ff5" Oct 31 13:30:03.468545 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:03.468438 4542 generic.go:356] "Generic (PLEG): container finished" podID="305d94b0-451d-446a-acdf-51f5acb18a15" containerID="28e71f7327a8dbd5cd8a6b8f11ec8f8b102e24d19c1cf1b25321028d6a8dad24" exitCode=60 Oct 31 13:30:03.468545 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:03.468502 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" event={"ID":"305d94b0-451d-446a-acdf-51f5acb18a15","Type":"ContainerDied","Data":"28e71f7327a8dbd5cd8a6b8f11ec8f8b102e24d19c1cf1b25321028d6a8dad24"} Oct 31 13:30:03.469083 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:03.468557 4542 scope.go:122] "RemoveContainer" containerID="03db179f834a47bdea97bf50ae295bd403f6abb3dace5f5f5558e04714fc0ff5" Oct 31 13:30:03.469083 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:03.468831 4542 scope.go:122] "RemoveContainer" containerID="28e71f7327a8dbd5cd8a6b8f11ec8f8b102e24d19c1cf1b25321028d6a8dad24" Oct 31 13:30:03.469192 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:30:03.469114 4542 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890890-wcbhj_odh-ai-gateway-infra(305d94b0-451d-446a-acdf-51f5acb18a15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" podUID="305d94b0-451d-446a-acdf-51f5acb18a15" Oct 31 13:30:04.474247 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:04.474217 4542 scope.go:122] "RemoveContainer" containerID="28e71f7327a8dbd5cd8a6b8f11ec8f8b102e24d19c1cf1b25321028d6a8dad24" Oct 31 13:30:04.474663 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:30:04.474441 4542 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890890-wcbhj_odh-ai-gateway-infra(305d94b0-451d-446a-acdf-51f5acb18a15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" podUID="305d94b0-451d-446a-acdf-51f5acb18a15" Oct 31 13:30:16.285280 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:16.285247 4542 scope.go:122] "RemoveContainer" containerID="28e71f7327a8dbd5cd8a6b8f11ec8f8b102e24d19c1cf1b25321028d6a8dad24" Oct 31 13:30:17.543410 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:17.543376 4542 generic.go:356] "Generic (PLEG): container finished" podID="305d94b0-451d-446a-acdf-51f5acb18a15" containerID="62b9433c574e16ed86b05f14c7518d143760167fd00f01f30a9461f2086919e6" exitCode=60 Oct 31 13:30:17.543856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:17.543450 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" event={"ID":"305d94b0-451d-446a-acdf-51f5acb18a15","Type":"ContainerDied","Data":"62b9433c574e16ed86b05f14c7518d143760167fd00f01f30a9461f2086919e6"} Oct 31 13:30:17.543856 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:17.543543 4542 scope.go:122] "RemoveContainer" containerID="28e71f7327a8dbd5cd8a6b8f11ec8f8b102e24d19c1cf1b25321028d6a8dad24" Oct 31 13:30:17.543966 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:17.543916 4542 scope.go:122] "RemoveContainer" containerID="62b9433c574e16ed86b05f14c7518d143760167fd00f01f30a9461f2086919e6" Oct 31 13:30:17.544214 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:30:17.544192 4542 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890890-wcbhj_odh-ai-gateway-infra(305d94b0-451d-446a-acdf-51f5acb18a15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" podUID="305d94b0-451d-446a-acdf-51f5acb18a15" Oct 31 13:30:18.575710 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:18.575677 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj"] Oct 31 13:30:18.713747 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:18.713722 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" Oct 31 13:30:18.785725 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:18.785689 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305d94b0-451d-446a-acdf-51f5acb18a15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305d94b0-451d-446a-acdf-51f5acb18a15-maas-api-service-ca\") pod \"305d94b0-451d-446a-acdf-51f5acb18a15\" (UID: \"305d94b0-451d-446a-acdf-51f5acb18a15\") " Oct 31 13:30:18.785917 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:18.785795 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305d94b0-451d-446a-acdf-51f5acb18a15-kube-api-access-k89sh\" (UniqueName: \"kubernetes.io/projected/305d94b0-451d-446a-acdf-51f5acb18a15-kube-api-access-k89sh\") pod \"305d94b0-451d-446a-acdf-51f5acb18a15\" (UID: \"305d94b0-451d-446a-acdf-51f5acb18a15\") " Oct 31 13:30:18.786068 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:18.786040 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305d94b0-451d-446a-acdf-51f5acb18a15-maas-api-service-ca" pod "305d94b0-451d-446a-acdf-51f5acb18a15" (UID: "305d94b0-451d-446a-acdf-51f5acb18a15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:18.787809 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:18.787778 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305d94b0-451d-446a-acdf-51f5acb18a15-kube-api-access-k89sh" pod "305d94b0-451d-446a-acdf-51f5acb18a15" (UID: "305d94b0-451d-446a-acdf-51f5acb18a15"). InnerVolumeSpecName "kube-api-access-k89sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:18.887386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:18.887296 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k89sh\" (UniqueName: \"kubernetes.io/projected/305d94b0-451d-446a-acdf-51f5acb18a15-kube-api-access-k89sh\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:18.887386 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:18.887335 4542 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305d94b0-451d-446a-acdf-51f5acb18a15-maas-api-service-ca\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:19.557756 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:19.557663 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" Oct 31 13:30:19.557756 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:19.557660 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj" event={"ID":"305d94b0-451d-446a-acdf-51f5acb18a15","Type":"ContainerDied","Data":"71cef74444c0aaca02060a3bbf868f92b92b9535e94f569eca4c5962486feeb2"} Oct 31 13:30:19.557984 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:19.557811 4542 scope.go:122] "RemoveContainer" containerID="62b9433c574e16ed86b05f14c7518d143760167fd00f01f30a9461f2086919e6" Oct 31 13:30:19.585152 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:19.585118 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj"] Oct 31 13:30:19.595117 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:19.595093 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-wcbhj"] Oct 31 13:30:21.290447 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:30:21.290413 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305d94b0-451d-446a-acdf-51f5acb18a15" path="/var/lib/kubelet/pods/305d94b0-451d-446a-acdf-51f5acb18a15/volumes" Oct 31 13:30:23.834559 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:30:23.834522 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:30:53.832010 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:30:53.831971 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:31:14.978876 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:14.978844 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg"] Oct 31 13:31:15.005805 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:15.005765 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4"] Oct 31 13:31:19.980087 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:19.980018 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" podUID="c6880566-9beb-4933-869a-4bc3ce78b016" containerName="payload-processing" containerID="cri-o://57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6" gracePeriod=25 Oct 31 13:31:20.006063 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.006023 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" podUID="f0e4e4ab-32eb-4a22-97d4-641d98f034a2" containerName="payload-pre-processing" containerID="cri-o://d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3" gracePeriod=25 Oct 31 13:31:20.262955 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.262929 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:31:20.287049 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.287008 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:31:20.368911 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.368881 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6880566-9beb-4933-869a-4bc3ce78b016-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6880566-9beb-4933-869a-4bc3ce78b016-plugins-config-volume\") pod \"c6880566-9beb-4933-869a-4bc3ce78b016\" (UID: \"c6880566-9beb-4933-869a-4bc3ce78b016\") " Oct 31 13:31:20.369080 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.368954 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6880566-9beb-4933-869a-4bc3ce78b016-kube-api-access-92zph\" (UniqueName: \"kubernetes.io/projected/c6880566-9beb-4933-869a-4bc3ce78b016-kube-api-access-92zph\") pod \"c6880566-9beb-4933-869a-4bc3ce78b016\" (UID: \"c6880566-9beb-4933-869a-4bc3ce78b016\") " Oct 31 13:31:20.369129 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.369112 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-plugins-config-volume\") pod \"f0e4e4ab-32eb-4a22-97d4-641d98f034a2\" (UID: \"f0e4e4ab-32eb-4a22-97d4-641d98f034a2\") " Oct 31 13:31:20.369196 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.369179 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-kube-api-access-whvxv\" (UniqueName: \"kubernetes.io/projected/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-kube-api-access-whvxv\") pod \"f0e4e4ab-32eb-4a22-97d4-641d98f034a2\" (UID: \"f0e4e4ab-32eb-4a22-97d4-641d98f034a2\") " Oct 31 13:31:20.369342 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.369321 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6880566-9beb-4933-869a-4bc3ce78b016-plugins-config-volume" pod "c6880566-9beb-4933-869a-4bc3ce78b016" (UID: "c6880566-9beb-4933-869a-4bc3ce78b016"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:20.369540 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.369518 4542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6880566-9beb-4933-869a-4bc3ce78b016-plugins-config-volume\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.369540 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.369525 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-plugins-config-volume" pod "f0e4e4ab-32eb-4a22-97d4-641d98f034a2" (UID: "f0e4e4ab-32eb-4a22-97d4-641d98f034a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:20.371078 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.371058 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6880566-9beb-4933-869a-4bc3ce78b016-kube-api-access-92zph" pod "c6880566-9beb-4933-869a-4bc3ce78b016" (UID: "c6880566-9beb-4933-869a-4bc3ce78b016"). InnerVolumeSpecName "kube-api-access-92zph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:20.371178 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.371100 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-kube-api-access-whvxv" pod "f0e4e4ab-32eb-4a22-97d4-641d98f034a2" (UID: "f0e4e4ab-32eb-4a22-97d4-641d98f034a2"). InnerVolumeSpecName "kube-api-access-whvxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:20.470408 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.470374 4542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-plugins-config-volume\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.470408 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.470402 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whvxv\" (UniqueName: \"kubernetes.io/projected/f0e4e4ab-32eb-4a22-97d4-641d98f034a2-kube-api-access-whvxv\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.470614 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.470415 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92zph\" (UniqueName: \"kubernetes.io/projected/c6880566-9beb-4933-869a-4bc3ce78b016-kube-api-access-92zph\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:20.853410 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.853371 4542 generic.go:356] "Generic (PLEG): container finished" podID="f0e4e4ab-32eb-4a22-97d4-641d98f034a2" containerID="d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3" exitCode=0 Oct 31 13:31:20.853598 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.853419 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" Oct 31 13:31:20.853598 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.853458 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" event={"ID":"f0e4e4ab-32eb-4a22-97d4-641d98f034a2","Type":"ContainerDied","Data":"d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3"} Oct 31 13:31:20.853598 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.853536 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4" event={"ID":"f0e4e4ab-32eb-4a22-97d4-641d98f034a2","Type":"ContainerDied","Data":"30f994ee1ae7427f9aa2a7e75e74807a945d33ccc40d06eceb4f2e39b91178db"} Oct 31 13:31:20.853598 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.853562 4542 scope.go:122] "RemoveContainer" containerID="d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3" Oct 31 13:31:20.855049 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.855020 4542 generic.go:356] "Generic (PLEG): container finished" podID="c6880566-9beb-4933-869a-4bc3ce78b016" containerID="57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6" exitCode=0 Oct 31 13:31:20.855183 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.855059 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" Oct 31 13:31:20.855183 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.855105 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" event={"ID":"c6880566-9beb-4933-869a-4bc3ce78b016","Type":"ContainerDied","Data":"57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6"} Oct 31 13:31:20.855183 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.855167 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg" event={"ID":"c6880566-9beb-4933-869a-4bc3ce78b016","Type":"ContainerDied","Data":"6cdfbbfbe7882abd73b81baff81703bbc3ae7319dbba847c2927f10d4fc497e7"} Oct 31 13:31:20.868564 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.868537 4542 scope.go:122] "RemoveContainer" containerID="d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3" Oct 31 13:31:20.869084 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:31:20.869062 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3\": container with ID starting with d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3 not found: ID does not exist" containerID="d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3" Oct 31 13:31:20.869172 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.869096 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3"} err="failed to get container status \"d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3\": rpc error: code = NotFound desc = could not find container \"d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3\": container with ID starting with d70eb5f2e169e4911e481d60c284489678e8a0ccfa3f0ff72ef68ea69570a5e3 not found: ID does not exist" Oct 31 13:31:20.869172 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.869115 4542 scope.go:122] "RemoveContainer" containerID="57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6" Oct 31 13:31:20.877180 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.877157 4542 scope.go:122] "RemoveContainer" containerID="57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6" Oct 31 13:31:20.877434 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:31:20.877416 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6\": container with ID starting with 57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6 not found: ID does not exist" containerID="57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6" Oct 31 13:31:20.877508 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.877442 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6"} err="failed to get container status \"57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6\": rpc error: code = NotFound desc = could not find container \"57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6\": container with ID starting with 57bc9eed32c7a58474dc60fba159de67c84e12fe687b4389bb701c509935a6f6 not found: ID does not exist" Oct 31 13:31:20.885752 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.885730 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg"] Oct 31 13:31:20.888222 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.888201 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9c81e5-5c658fbdfc-gvkkg"] Oct 31 13:31:20.900634 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.900614 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4"] Oct 31 13:31:20.905743 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:20.905715 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9c81e5-d6777c5f-mhgt4"] Oct 31 13:31:21.292730 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:21.292648 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6880566-9beb-4933-869a-4bc3ce78b016" path="/var/lib/kubelet/pods/c6880566-9beb-4933-869a-4bc3ce78b016/volumes" Oct 31 13:31:21.293140 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:21.293112 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e4e4ab-32eb-4a22-97d4-641d98f034a2" path="/var/lib/kubelet/pods/f0e4e4ab-32eb-4a22-97d4-641d98f034a2/volumes" Oct 31 13:31:23.051558 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:23.051522 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:31:23.833890 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:31:23.833854 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:31:29.217994 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:29.217961 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8"] Oct 31 13:31:29.218452 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:29.218187 4542 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" containerID="cri-o://18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518" gracePeriod=30 Oct 31 13:31:31.327310 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:31.327280 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:31:34.615694 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.615661 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:31:34.747175 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747144 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7470834a-3540-460d-ac9b-5133bd2c7132-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7470834a-3540-460d-ac9b-5133bd2c7132-istio-podinfo\") pod \"7470834a-3540-460d-ac9b-5133bd2c7132\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " Oct 31 13:31:34.747366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747196 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-istio-token\" (UniqueName: \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-istio-token\") pod \"7470834a-3540-460d-ac9b-5133bd2c7132\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " Oct 31 13:31:34.747366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747219 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-envoy\") pod \"7470834a-3540-460d-ac9b-5133bd2c7132\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " Oct 31 13:31:34.747366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747250 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-kube-api-access-hc2lz\" (UniqueName: \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-kube-api-access-hc2lz\") pod \"7470834a-3540-460d-ac9b-5133bd2c7132\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " Oct 31 13:31:34.747366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747279 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-certs\") pod \"7470834a-3540-460d-ac9b-5133bd2c7132\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " Oct 31 13:31:34.747366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747302 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-data\") pod \"7470834a-3540-460d-ac9b-5133bd2c7132\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " Oct 31 13:31:34.747366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747331 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-socket\") pod \"7470834a-3540-460d-ac9b-5133bd2c7132\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " Oct 31 13:31:34.747366 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747368 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-credential-socket\") pod \"7470834a-3540-460d-ac9b-5133bd2c7132\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " Oct 31 13:31:34.747817 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747404 4542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7470834a-3540-460d-ac9b-5133bd2c7132-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7470834a-3540-460d-ac9b-5133bd2c7132-istiod-ca-cert\") pod \"7470834a-3540-460d-ac9b-5133bd2c7132\" (UID: \"7470834a-3540-460d-ac9b-5133bd2c7132\") " Oct 31 13:31:34.747817 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747676 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-data" pod "7470834a-3540-460d-ac9b-5133bd2c7132" (UID: "7470834a-3540-460d-ac9b-5133bd2c7132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:34.747817 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747752 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-socket" pod "7470834a-3540-460d-ac9b-5133bd2c7132" (UID: "7470834a-3540-460d-ac9b-5133bd2c7132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:34.747817 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747740 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-certs" pod "7470834a-3540-460d-ac9b-5133bd2c7132" (UID: "7470834a-3540-460d-ac9b-5133bd2c7132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:34.748031 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747837 4542 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-certs\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.748031 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747856 4542 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-data\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.748031 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.747870 4542 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-workload-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.748031 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.748018 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7470834a-3540-460d-ac9b-5133bd2c7132-istiod-ca-cert" pod "7470834a-3540-460d-ac9b-5133bd2c7132" (UID: "7470834a-3540-460d-ac9b-5133bd2c7132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:34.748240 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.748092 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-credential-socket" pod "7470834a-3540-460d-ac9b-5133bd2c7132" (UID: "7470834a-3540-460d-ac9b-5133bd2c7132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:34.749582 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.749549 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7470834a-3540-460d-ac9b-5133bd2c7132-istio-podinfo" pod "7470834a-3540-460d-ac9b-5133bd2c7132" (UID: "7470834a-3540-460d-ac9b-5133bd2c7132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:34.749938 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.749904 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-kube-api-access-hc2lz" pod "7470834a-3540-460d-ac9b-5133bd2c7132" (UID: "7470834a-3540-460d-ac9b-5133bd2c7132"). InnerVolumeSpecName "kube-api-access-hc2lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:34.750075 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.750056 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-istio-token" pod "7470834a-3540-460d-ac9b-5133bd2c7132" (UID: "7470834a-3540-460d-ac9b-5133bd2c7132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:34.750118 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.750070 4542 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-envoy" pod "7470834a-3540-460d-ac9b-5133bd2c7132" (UID: "7470834a-3540-460d-ac9b-5133bd2c7132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:34.849122 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.849099 4542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2lz\" (UniqueName: \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-kube-api-access-hc2lz\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.849122 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.849122 4542 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-credential-socket\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.849292 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.849139 4542 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7470834a-3540-460d-ac9b-5133bd2c7132-istiod-ca-cert\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.849292 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.849150 4542 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7470834a-3540-460d-ac9b-5133bd2c7132-istio-podinfo\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.849292 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.849158 4542 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7470834a-3540-460d-ac9b-5133bd2c7132-istio-token\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.849292 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.849166 4542 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7470834a-3540-460d-ac9b-5133bd2c7132-istio-envoy\") on node \"ip-10-0-137-206.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.954904 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.954856 4542 generic.go:356] "Generic (PLEG): container finished" podID="7470834a-3540-460d-ac9b-5133bd2c7132" containerID="18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518" exitCode=0 Oct 31 13:31:34.955091 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.954908 4542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" Oct 31 13:31:34.955091 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.954943 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" event={"ID":"7470834a-3540-460d-ac9b-5133bd2c7132","Type":"ContainerDied","Data":"18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518"} Oct 31 13:31:34.955091 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.955018 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8" event={"ID":"7470834a-3540-460d-ac9b-5133bd2c7132","Type":"ContainerDied","Data":"c7a4c74be12e5263bd0854763b240d1911e8723f2108637c06a2ddb7eaa17262"} Oct 31 13:31:34.955091 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.955040 4542 scope.go:122] "RemoveContainer" containerID="18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518" Oct 31 13:31:34.964377 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.964362 4542 scope.go:122] "RemoveContainer" containerID="a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7" Oct 31 13:31:34.972100 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.972086 4542 scope.go:122] "RemoveContainer" containerID="18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518" Oct 31 13:31:34.972348 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:31:34.972334 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518\": container with ID starting with 18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518 not found: ID does not exist" containerID="18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518" Oct 31 13:31:34.972396 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.972356 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518"} err="failed to get container status \"18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518\": rpc error: code = NotFound desc = could not find container \"18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518\": container with ID starting with 18b5bd0d0b982df6b527131259a7ec8536909a7dd2652191d718e1e602aa4518 not found: ID does not exist" Oct 31 13:31:34.972396 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.972370 4542 scope.go:122] "RemoveContainer" containerID="a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7" Oct 31 13:31:34.972660 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:31:34.972641 4542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7\": container with ID starting with a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7 not found: ID does not exist" containerID="a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7" Oct 31 13:31:34.972711 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.972666 4542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7"} err="failed to get container status \"a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7\": rpc error: code = NotFound desc = could not find container \"a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7\": container with ID starting with a4911a082334ca2d4b131b9e204d361d583935caea81a02eb6b2462b6bfb04b7 not found: ID does not exist" Oct 31 13:31:34.982026 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.982006 4542 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8"] Oct 31 13:31:34.992523 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:34.992504 4542 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9c81e5-openshift-default-774d88b6fc-v6kr8"] Oct 31 13:31:35.289980 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:35.289902 4542 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" path="/var/lib/kubelet/pods/7470834a-3540-460d-ac9b-5133bd2c7132/volumes" Oct 31 13:31:36.528172 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:36.528137 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:31:51.894000 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:31:51.893963 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:31:53.831906 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:31:53.831866 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:32:18.729423 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:18.729386 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hptgn"] Oct 31 13:32:23.831892 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:32:23.831851 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:32:30.136914 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:30.136882 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-88dcl_9dd88af6-332b-4505-8816-ac41c1421fb9/manager/0.log" Oct 31 13:32:30.775811 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:30.775786 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7n9z4_b7ac2673-e6fe-4c31-b278-64345475da61/server/0.log" Oct 31 13:32:31.882189 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:31.882158 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7df6dd5c76-crvv7_fa049032-484c-43c5-9440-7928760a0967/maas-api/0.log" Oct 31 13:32:35.804219 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:35.804186 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hptgn_3c604ceb-6d74-416f-924d-22a8f5e835d7/limitador/0.log" Oct 31 13:32:35.989266 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:35.989231 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-c6vhc_31c91a14-a352-471c-9857-d483ac2c758a/manager/0.log" Oct 31 13:32:37.106412 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:37.106384 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-776d875d45-6shpq_f3080f7f-0fbf-40eb-9f27-89928be76699/kube-auth-proxy/0.log" Oct 31 13:32:37.656214 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:37.656186 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-5v9c9_ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc/istio-proxy/0.log" Oct 31 13:32:38.576851 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:38.576817 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-nq8qc_9ec44fe2-a89f-4ae6-8f6f-7220557b19b1/istio-proxy/0.log" Oct 31 13:32:38.938832 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:38.938748 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vl2sm_ce755ffc-b2d4-45c2-9054-6022e18db453/istio-proxy/0.log" Oct 31 13:32:39.116134 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:39.116108 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-xvwjv_3defcda2-f67c-4057-b3c4-ec130bbd4e91/istio-proxy/0.log" Oct 31 13:32:40.772274 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:40.772249 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd_a4599d10-3cf5-45df-bb98-b52d2911367b/main/0.log" Oct 31 13:32:41.134767 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:41.134740 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xtnsg_fda89da9-cda2-4148-ae7b-70f8966b3769/main/0.log" Oct 31 13:32:41.495047 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:41.494971 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg_8269080b-1265-480e-a205-976decbd5bfa/main/0.log" Oct 31 13:32:42.047725 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:42.047695 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj_1d5dc13e-a93d-4867-bd22-ccf73246c370/main/0.log" Oct 31 13:32:52.471430 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:52.471392 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-88dcl_9dd88af6-332b-4505-8816-ac41c1421fb9/manager/0.log" Oct 31 13:32:53.095334 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:53.095310 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7n9z4_b7ac2673-e6fe-4c31-b278-64345475da61/server/0.log" Oct 31 13:32:53.830950 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:32:53.830914 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:32:54.052981 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:54.052946 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7df6dd5c76-crvv7_fa049032-484c-43c5-9440-7928760a0967/maas-api/0.log" Oct 31 13:32:57.473263 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:57.473235 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hptgn_3c604ceb-6d74-416f-924d-22a8f5e835d7/limitador/0.log" Oct 31 13:32:57.662415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:57.662386 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-c6vhc_31c91a14-a352-471c-9857-d483ac2c758a/manager/0.log" Oct 31 13:32:58.618820 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:58.618790 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-776d875d45-6shpq_f3080f7f-0fbf-40eb-9f27-89928be76699/kube-auth-proxy/0.log" Oct 31 13:32:59.160899 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:32:59.160873 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-5v9c9_ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc/istio-proxy/0.log" Oct 31 13:33:00.104918 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:33:00.104886 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-nq8qc_9ec44fe2-a89f-4ae6-8f6f-7220557b19b1/istio-proxy/0.log" Oct 31 13:33:00.479202 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:33:00.479123 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vl2sm_ce755ffc-b2d4-45c2-9054-6022e18db453/istio-proxy/0.log" Oct 31 13:33:00.668490 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:33:00.668448 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-xvwjv_3defcda2-f67c-4057-b3c4-ec130bbd4e91/istio-proxy/0.log" Oct 31 13:33:01.921205 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:33:01.921176 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zxxdd_a4599d10-3cf5-45df-bb98-b52d2911367b/main/0.log" Oct 31 13:33:02.280415 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:33:02.280348 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xtnsg_fda89da9-cda2-4148-ae7b-70f8966b3769/main/0.log" Oct 31 13:33:02.636124 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:33:02.636092 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m78rg_8269080b-1265-480e-a205-976decbd5bfa/main/0.log" Oct 31 13:33:03.182932 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:33:03.182908 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-64cnj_1d5dc13e-a93d-4867-bd22-ccf73246c370/main/0.log" Oct 31 13:33:23.830907 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:33:23.830856 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:33:53.831094 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:33:53.831059 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:34:23.830858 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:34:23.830813 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:34:40.192653 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:40.192610 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w58sk_d83f70c7-ab6c-43b0-b86d-3d9d935973ec/global-pull-secret-syncer/0.log" Oct 31 13:34:40.453232 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:40.453157 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rwtn6_1aaae2c3-0ea2-4061-a448-e50f82dcf823/konnectivity-agent/0.log" Oct 31 13:34:40.509135 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:40.509106 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-206.us-east-2.compute.internal_df9f494cd77ee5cda911dbf4d5fd20aa/haproxy/0.log" Oct 31 13:34:43.752714 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:43.752686 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:34:43.759316 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:43.759289 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbk8s_fa9e011c-a51f-4738-b970-58ec77045b81/ovn-acl-logging/0.log" Oct 31 13:34:47.890591 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:47.890555 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hptgn_3c604ceb-6d74-416f-924d-22a8f5e835d7/limitador/0.log" Oct 31 13:34:47.956027 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:47.955990 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-c6vhc_31c91a14-a352-471c-9857-d483ac2c758a/manager/0.log" Oct 31 13:34:49.671729 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:49.671699 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7d4p_417e2a85-64e8-451c-b6d9-f553c70d6fe8/node-exporter/0.log" Oct 31 13:34:49.706622 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:49.706598 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7d4p_417e2a85-64e8-451c-b6d9-f553c70d6fe8/kube-rbac-proxy/0.log" Oct 31 13:34:49.734044 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:49.734023 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7d4p_417e2a85-64e8-451c-b6d9-f553c70d6fe8/init-textfile/0.log" Oct 31 13:34:49.955829 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:49.955761 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4b3ffd0-32aa-4fb0-b3a4-191895c46215/prometheus/0.log" Oct 31 13:34:49.983044 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:49.983022 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4b3ffd0-32aa-4fb0-b3a4-191895c46215/config-reloader/0.log" Oct 31 13:34:50.011738 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.011718 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4b3ffd0-32aa-4fb0-b3a4-191895c46215/thanos-sidecar/0.log" Oct 31 13:34:50.041287 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.041269 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4b3ffd0-32aa-4fb0-b3a4-191895c46215/kube-rbac-proxy-web/0.log" Oct 31 13:34:50.073636 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.073617 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4b3ffd0-32aa-4fb0-b3a4-191895c46215/kube-rbac-proxy/0.log" Oct 31 13:34:50.101692 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.101674 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4b3ffd0-32aa-4fb0-b3a4-191895c46215/kube-rbac-proxy-thanos/0.log" Oct 31 13:34:50.131223 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.131205 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f4b3ffd0-32aa-4fb0-b3a4-191895c46215/init-config-reloader/0.log" Oct 31 13:34:50.239092 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.239030 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lc8jp_1d238636-8cb3-4923-b4b0-d58909ef45d2/prometheus-operator-admission-webhook/0.log" Oct 31 13:34:50.310959 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.310929 4542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt"] Oct 31 13:34:50.311284 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311272 4542 cpu_manager.go:414] "containerMap: removing container" podUID="305d94b0-451d-446a-acdf-51f5acb18a15" containerName="cleanup" Oct 31 13:34:50.311327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311285 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305d94b0-451d-446a-acdf-51f5acb18a15" containerName="cleanup" Oct 31 13:34:50.311327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311295 4542 cpu_manager.go:414] "containerMap: removing container" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" Oct 31 13:34:50.311327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311300 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" Oct 31 13:34:50.311327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311316 4542 cpu_manager.go:414] "containerMap: removing container" podUID="305d94b0-451d-446a-acdf-51f5acb18a15" containerName="cleanup" Oct 31 13:34:50.311327 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311322 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305d94b0-451d-446a-acdf-51f5acb18a15" containerName="cleanup" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311336 4542 cpu_manager.go:414] "containerMap: removing container" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311341 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311353 4542 cpu_manager.go:414] "containerMap: removing container" podUID="f0e4e4ab-32eb-4a22-97d4-641d98f034a2" containerName="payload-pre-processing" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311359 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e4e4ab-32eb-4a22-97d4-641d98f034a2" containerName="payload-pre-processing" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311371 4542 cpu_manager.go:414] "containerMap: removing container" podUID="c6880566-9beb-4933-869a-4bc3ce78b016" containerName="payload-processing" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311375 4542 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6880566-9beb-4933-869a-4bc3ce78b016" containerName="payload-processing" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311427 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="305d94b0-451d-446a-acdf-51f5acb18a15" containerName="cleanup" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311434 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e4e4ab-32eb-4a22-97d4-641d98f034a2" containerName="payload-pre-processing" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311441 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6880566-9beb-4933-869a-4bc3ce78b016" containerName="payload-processing" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311448 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="305d94b0-451d-446a-acdf-51f5acb18a15" containerName="cleanup" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311453 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" Oct 31 13:34:50.311500 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.311461 4542 memory_manager.go:365] "RemoveStaleState removing state" podUID="7470834a-3540-460d-ac9b-5133bd2c7132" containerName="istio-proxy" Oct 31 13:34:50.314967 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.314947 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.327412 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.327329 4542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8xtmm\"/\"default-dockercfg-c9924\"" Oct 31 13:34:50.328650 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.328627 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt"] Oct 31 13:34:50.419618 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.419591 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xtmm\"/\"openshift-service-ca.crt\"" Oct 31 13:34:50.419618 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.419614 4542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xtmm\"/\"kube-root-ca.crt\"" Oct 31 13:34:50.467360 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.467331 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9nfb\" (UniqueName: \"kubernetes.io/projected/1bf63627-19b2-408f-a190-7e5ff71296bb-kube-api-access-h9nfb\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.467502 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.467367 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-sys\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.467594 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.467572 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-proc\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.467685 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.467666 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-lib-modules\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.467798 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.467777 4542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-podres\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.568962 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.568933 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-proc\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.569074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.568982 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-lib-modules\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.569074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.569014 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-podres\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.569074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.569041 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-proc\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.569074 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.569058 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9nfb\" (UniqueName: \"kubernetes.io/projected/1bf63627-19b2-408f-a190-7e5ff71296bb-kube-api-access-h9nfb\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.569301 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.569080 4542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-sys\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.569301 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.569105 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-podres\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.569301 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.569123 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-lib-modules\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.569301 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.569186 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bf63627-19b2-408f-a190-7e5ff71296bb-sys\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.582363 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.582338 4542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9nfb\" (UniqueName: \"kubernetes.io/projected/1bf63627-19b2-408f-a190-7e5ff71296bb-kube-api-access-h9nfb\") pod \"perf-node-gather-daemonset-zr4nt\" (UID: \"1bf63627-19b2-408f-a190-7e5ff71296bb\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.627913 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.627889 4542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:50.901691 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:50.901602 4542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt"] Oct 31 13:34:50.904629 ip-10-0-137-206 kubenswrapper[4542]: W1031 13:34:50.904592 4542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1bf63627_19b2_408f_a190_7e5ff71296bb.slice/crio-34b9ba1f39d45026b96d6a119ce92b797cc551fd5bfa922d890d903aeeefc593 WatchSource:0}: Error finding container 34b9ba1f39d45026b96d6a119ce92b797cc551fd5bfa922d890d903aeeefc593: Status 404 returned error can't find the container with id 34b9ba1f39d45026b96d6a119ce92b797cc551fd5bfa922d890d903aeeefc593 Oct 31 13:34:51.863093 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:51.863053 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" event={"ID":"1bf63627-19b2-408f-a190-7e5ff71296bb","Type":"ContainerStarted","Data":"e26cc4c8223d56a1b7a2f917852d535fb8d1441542921de4042f7b6b0139b834"} Oct 31 13:34:51.863294 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:51.863189 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:51.863294 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:51.863230 4542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" event={"ID":"1bf63627-19b2-408f-a190-7e5ff71296bb","Type":"ContainerStarted","Data":"34b9ba1f39d45026b96d6a119ce92b797cc551fd5bfa922d890d903aeeefc593"} Oct 31 13:34:51.884942 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:51.884899 4542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" podStartSLOduration=1.88488704 podStartE2EDuration="1.88488704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:51.882624914 +0000 UTC m=+2108.727791333" watchObservedRunningTime="2026-10-31 13:34:51.88488704 +0000 UTC m=+2108.730053458" Oct 31 13:34:52.988523 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:52.988470 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-79n4w_cd02e70d-668b-42d9-928e-ca8ddfd352d0/download-server/0.log" Oct 31 13:34:53.834500 ip-10-0-137-206 kubenswrapper[4542]: E1031 13:34:53.834447 4542 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c2knk" containerName="extractor" Oct 31 13:34:55.014336 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:55.014306 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ltf7k_f338bc5c-18e1-4e62-be2d-93de38d4ae9d/dns/0.log" Oct 31 13:34:55.042146 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:55.042112 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ltf7k_f338bc5c-18e1-4e62-be2d-93de38d4ae9d/kube-rbac-proxy/0.log" Oct 31 13:34:55.133230 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:55.133203 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cs9pm_ed98cbbd-063e-43ed-bddc-ba216fbf56bb/dns-node-resolver/0.log" Oct 31 13:34:55.898852 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:55.898814 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84f677497d-dp9bl_09ac04c7-a98d-422d-8f55-3a1c4a41d952/registry/0.log" Oct 31 13:34:55.984810 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:55.984784 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s7fjc_5d525238-6ced-4f9f-89a6-56fb50fba909/node-ca/0.log" Oct 31 13:34:57.878016 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:57.877937 4542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-zr4nt" Oct 31 13:34:58.258032 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:58.257955 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-776d875d45-6shpq_f3080f7f-0fbf-40eb-9f27-89928be76699/kube-auth-proxy/0.log" Oct 31 13:34:58.360696 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:58.360666 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-5v9c9_ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc/istio-proxy/0.log" Oct 31 13:34:58.529978 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:58.529903 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-nq8qc_9ec44fe2-a89f-4ae6-8f6f-7220557b19b1/istio-proxy/0.log" Oct 31 13:34:58.599017 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:58.598990 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vl2sm_ce755ffc-b2d4-45c2-9054-6022e18db453/istio-proxy/0.log" Oct 31 13:34:58.630924 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:34:58.630899 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-xvwjv_3defcda2-f67c-4057-b3c4-ec130bbd4e91/istio-proxy/0.log" Oct 31 13:35:00.381095 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:00.381055 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rm7b9_9e16e028-6e02-48e1-b54e-3517232c2d8b/serve-healthcheck-canary/0.log" Oct 31 13:35:01.190971 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:01.190945 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2knk_7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400/kube-rbac-proxy/0.log" Oct 31 13:35:01.250938 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:01.250906 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2knk_7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400/exporter/0.log" Oct 31 13:35:01.311873 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:01.311826 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2knk_7e2cb3ac-95ac-4d5b-96ac-cd6e2c382400/extractor/0.log" Oct 31 13:35:02.170417 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:02.170386 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-776d875d45-6shpq_f3080f7f-0fbf-40eb-9f27-89928be76699/kube-auth-proxy/0.log" Oct 31 13:35:02.234610 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:02.234583 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-5v9c9_ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc/istio-proxy/0.log" Oct 31 13:35:02.346099 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:02.346076 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-nq8qc_9ec44fe2-a89f-4ae6-8f6f-7220557b19b1/istio-proxy/0.log" Oct 31 13:35:02.385236 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:02.385210 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vl2sm_ce755ffc-b2d4-45c2-9054-6022e18db453/istio-proxy/0.log" Oct 31 13:35:02.407243 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:02.407224 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-xvwjv_3defcda2-f67c-4057-b3c4-ec130bbd4e91/istio-proxy/0.log" Oct 31 13:35:04.310682 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:04.310653 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-849556f89d-9n7pr_4ca46985-7193-421f-8eb3-163824750a95/manager/0.log" Oct 31 13:35:04.339604 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:04.339580 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-849556f89d-9n7pr_4ca46985-7193-421f-8eb3-163824750a95/copy-manifests/0.log" Oct 31 13:35:05.826330 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:05.826301 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-776d875d45-6shpq_f3080f7f-0fbf-40eb-9f27-89928be76699/kube-auth-proxy/0.log" Oct 31 13:35:05.895873 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:05.895832 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-5v9c9_ade2f44e-b0bb-4004-a5c6-3d84e9e80fcc/istio-proxy/0.log" Oct 31 13:35:06.004982 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:06.004955 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-nq8qc_9ec44fe2-a89f-4ae6-8f6f-7220557b19b1/istio-proxy/0.log" Oct 31 13:35:06.048155 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:06.048121 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vl2sm_ce755ffc-b2d4-45c2-9054-6022e18db453/istio-proxy/0.log" Oct 31 13:35:06.070645 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:06.070617 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-xvwjv_3defcda2-f67c-4057-b3c4-ec130bbd4e91/istio-proxy/0.log" Oct 31 13:35:06.496659 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:06.496625 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-88dcl_9dd88af6-332b-4505-8816-ac41c1421fb9/manager/0.log" Oct 31 13:35:06.874550 ip-10-0-137-206 kubenswrapper[4542]: I1031 13:35:06.874521 4542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7n9z4_b7ac2673-e6fe-4c31-b278-64345475da61/server/0.log"