Oct 30 19:17:53.856210 ip-10-0-138-21 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 19:17:53.856222 ip-10-0-138-21 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 19:17:53.856232 ip-10-0-138-21 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 19:17:53.856557 ip-10-0-138-21 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 19:18:03.982535 ip-10-0-138-21 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 19:18:03.982550 ip-10-0-138-21 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b3f484a775f24fea87f683eac4cf4b49 -- Oct 30 19:19:57.022350 ip-10-0-138-21 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:19:57.374624 ip-10-0-138-21 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:19:57.374624 ip-10-0-138-21 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:19:57.374624 ip-10-0-138-21 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:19:57.374624 ip-10-0-138-21 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:19:57.381210 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381140 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:57.381210 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381208 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:57.381210 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381212 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:57.381210 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381215 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381219 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381222 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381225 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381227 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381230 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381232 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381235 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381238 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381241 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381243 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381246 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381249 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381251 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381254 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381257 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381259 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381262 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381265 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381267 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:57.381366 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381270 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381272 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381274 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381277 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381280 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381282 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381285 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381293 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381296 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381299 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381302 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381304 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381307 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381310 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381313 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381315 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381318 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381320 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381323 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381326 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:57.381827 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381330 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381335 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381339 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381342 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381345 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381348 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381351 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381353 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381356 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381359 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381362 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381365 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381368 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381370 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381373 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381376 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381378 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381381 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381383 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:57.382311 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381387 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381390 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381392 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381396 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381400 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381403 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381406 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381409 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381412 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381414 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381417 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381420 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381422 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381425 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381427 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381430 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381433 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381436 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381438 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:57.382755 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381441 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381443 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381447 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381450 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381452 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381455 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381458 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381460 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381463 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381465 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381467 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381470 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381472 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381475 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381477 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381480 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381482 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381485 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381487 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:57.383215 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381490 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381493 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381497 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381500 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381503 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.381506 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382361 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382368 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382371 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382374 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382377 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382379 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382382 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382384 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382387 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382390 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382393 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382395 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382398 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382400 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:57.383686 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382403 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382405 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382408 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382410 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382413 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382416 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382418 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382421 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382423 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382426 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382429 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382432 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382435 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382438 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382441 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382443 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382446 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382449 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382451 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:57.384182 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382455 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382460 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382463 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382465 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382468 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382470 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382473 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382475 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382478 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382480 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382484 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382486 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382489 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382491 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382494 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382496 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382499 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382501 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382504 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:57.384628 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382506 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382509 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382512 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382515 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382517 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382520 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382524 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382526 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382529 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382531 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382534 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382536 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382538 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382541 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382543 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382546 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382548 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382551 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382554 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382556 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:57.385076 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382559 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382561 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382564 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382566 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382570 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382572 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382575 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382577 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382580 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382582 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382585 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382588 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382590 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382593 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382597 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382599 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382602 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382605 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382607 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382609 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:57.385558 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382612 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382616 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382620 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382623 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382626 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382629 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382631 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382633 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382636 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382638 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382641 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382643 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382646 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.382648 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383176 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383183 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383187 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383190 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383193 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:57.386039 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383196 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383198 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383201 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383204 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383207 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383210 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383213 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383216 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383219 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383222 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383224 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383227 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383230 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383232 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383235 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383238 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383240 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383243 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383245 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:57.386491 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383248 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383250 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383253 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383256 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383259 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383262 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383265 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383268 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383270 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383273 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383275 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383278 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383281 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383284 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383286 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383289 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383293 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383296 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383298 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:57.386933 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383301 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383304 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383307 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383309 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383312 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383314 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383317 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383319 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383322 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383325 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383328 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383330 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383333 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383335 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383338 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383340 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383343 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383345 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383348 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383351 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:57.387426 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383354 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383356 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383359 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383361 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383364 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383367 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383371 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383375 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383378 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383383 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383386 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383388 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383391 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383394 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383396 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383398 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383401 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383404 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383406 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:57.387886 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383409 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383411 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383413 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383416 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383419 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383421 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383424 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383426 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383429 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383431 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383434 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383436 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383439 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383441 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383444 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383446 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383449 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383451 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383454 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383458 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:57.388392 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383461 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383464 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383468 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.383470 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384427 2573 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384441 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384448 2573 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384453 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384457 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384460 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384465 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384469 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384472 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384475 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384479 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384483 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384486 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384489 2573 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384492 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384495 2573 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384498 2573 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384501 2573 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384505 2573 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384508 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:19:57.388858 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384511 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384514 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384517 2573 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384521 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384524 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384527 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384530 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384533 2573 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384537 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384539 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384543 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384546 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384551 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384554 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384557 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384559 2573 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384563 2573 flags.go:64] FLAG: --enable-server="true" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384565 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384570 2573 flags.go:64] FLAG: --event-burst="100" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384574 2573 flags.go:64] FLAG: --event-qps="50" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384576 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384579 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384582 2573 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384586 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:19:57.389415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384589 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384592 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384595 2573 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384598 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384601 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384605 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384607 2573 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384610 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384613 2573 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384616 2573 flags.go:64] FLAG: --feature-gates="" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384620 2573 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384623 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384626 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384630 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384633 2573 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384636 2573 flags.go:64] FLAG: --help="false" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384639 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384643 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384646 2573 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384649 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384653 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384656 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384659 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:19:57.389982 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384662 2573 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384664 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384667 2573 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384670 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384673 2573 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384676 2573 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384679 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384682 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384685 2573 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384688 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384690 2573 flags.go:64] FLAG: --lock-file="" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384693 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384696 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384699 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384706 2573 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384709 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384712 2573 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384715 2573 flags.go:64] FLAG: --logging-format="text" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384718 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384721 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384724 2573 flags.go:64] FLAG: --manifest-url="" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384727 2573 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384731 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384734 2573 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384738 2573 flags.go:64] FLAG: --max-pods="110" Oct 30 19:19:57.390605 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384741 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384744 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384747 2573 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384750 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384753 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384756 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384759 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384766 2573 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384770 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384773 2573 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384775 2573 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384778 2573 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384781 2573 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384784 2573 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384787 2573 flags.go:64] FLAG: --port="10250" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384790 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384792 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02294910dd9115a67" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384795 2573 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384798 2573 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384801 2573 flags.go:64] FLAG: --register-node="true" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384804 2573 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384809 2573 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384812 2573 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384814 2573 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384817 2573 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384822 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:19:57.391247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384825 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384828 2573 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384831 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384833 2573 flags.go:64] FLAG: --runonce="false" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384836 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384839 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384842 2573 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384845 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384848 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384851 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384855 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384858 2573 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384860 2573 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384863 2573 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384866 2573 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384869 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384872 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384875 2573 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384877 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384882 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384885 2573 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384888 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384892 2573 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384894 2573 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384897 2573 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:19:57.391877 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384900 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384903 2573 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384906 2573 flags.go:64] FLAG: --v="2" Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384911 2573 flags.go:64] FLAG: --version="false" Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384915 2573 flags.go:64] FLAG: --vmodule="" Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384918 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.384922 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385075 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385079 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385082 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385085 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385088 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385093 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385097 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385117 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385120 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385122 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385126 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385129 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385131 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:57.392490 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385134 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385136 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385139 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385141 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385144 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385146 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385149 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385151 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385154 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385156 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385159 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385161 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385164 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385166 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385170 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385173 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385175 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385178 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385180 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385183 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:57.392968 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385185 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385188 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385190 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385193 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385197 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385199 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385202 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385205 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385207 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385210 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385213 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385215 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385218 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385220 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385223 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385225 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385228 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385230 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385233 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385235 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:57.393561 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385238 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385241 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385243 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385246 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385249 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385251 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385255 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385258 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385260 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385263 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385265 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385268 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385271 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385273 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385276 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385278 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385282 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385285 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385287 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385290 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:57.394025 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385292 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385295 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385298 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385308 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385311 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385314 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385316 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385318 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385321 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385323 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385326 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385329 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385331 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385334 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385336 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385339 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385341 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385344 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385347 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:57.394519 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385350 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385352 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385355 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385357 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385360 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385364 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385368 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385371 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385374 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385377 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385380 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385383 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385385 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385388 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.385415 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:19:57.394971 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385465 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385470 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385475 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385478 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385481 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385484 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385486 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385489 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385491 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385494 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385496 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385499 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385502 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385504 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385507 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385509 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385513 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385515 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385518 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385520 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:57.395338 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385523 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385526 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385528 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385531 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385533 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385536 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385538 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385542 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385544 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385547 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385550 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385552 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385555 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385557 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385560 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385563 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385566 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385569 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385572 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:57.395810 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385574 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385577 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385579 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385581 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385584 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385587 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385589 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385592 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385594 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385598 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385601 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385603 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385606 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385608 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385611 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385613 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385616 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385619 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385621 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385624 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:57.396275 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385628 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385631 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385633 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385636 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385639 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385641 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385644 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385646 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385649 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385652 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385654 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385657 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385659 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385662 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385664 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385667 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385669 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385672 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385676 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:57.396736 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385679 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385683 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385687 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385690 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385693 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385695 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385697 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385700 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385703 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385705 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385708 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385711 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385713 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385719 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385723 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385726 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385729 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385732 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385734 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:57.397198 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385737 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385739 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385742 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385744 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385747 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385750 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385753 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385755 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.385757 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.392434 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.392447 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.393055 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.393061 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393144 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393150 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393154 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393157 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393160 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393163 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:57.397671 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393165 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393168 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393171 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393173 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393176 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393178 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393180 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393184 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393186 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393189 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393191 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393194 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393196 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393199 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393202 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393204 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393207 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393210 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393213 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393216 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:57.398117 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393219 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393221 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393224 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393226 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393229 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393231 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393234 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393236 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393239 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393242 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393245 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393247 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393250 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393252 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393255 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393258 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393261 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393263 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393265 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:57.398591 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393268 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393271 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393273 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393278 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393282 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393285 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393288 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393290 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393293 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393295 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393298 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393301 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393304 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393307 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393311 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393314 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393317 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393320 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393323 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393325 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:57.399054 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393328 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393330 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393333 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393335 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393338 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393341 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393343 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393346 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393348 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393351 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393353 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393356 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393358 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393361 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393363 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393366 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393368 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393371 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393373 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393376 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:57.399533 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393378 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393381 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393383 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393386 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393389 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393391 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393394 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393396 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393399 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393402 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393404 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393407 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393409 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393412 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393415 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393417 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393420 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393422 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393425 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:57.399997 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393427 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393430 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.393457 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393613 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393617 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393620 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393623 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393626 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393628 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393631 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393634 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393636 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393639 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393641 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393644 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393646 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:57.400507 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393648 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393651 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393654 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393656 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393659 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393662 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393664 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393667 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393669 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393672 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393674 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393677 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393679 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393681 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393684 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393687 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393690 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393710 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393713 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393716 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:57.400896 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393719 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393722 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393724 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393728 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393732 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393735 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393738 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393741 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393743 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393746 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393748 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393751 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393754 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393756 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393758 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393762 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393766 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393769 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:57.401469 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393772 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393774 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393777 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393779 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393782 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393784 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393787 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393789 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393792 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393795 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393797 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393799 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393802 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393804 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393809 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393812 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393814 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393817 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393819 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393822 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:57.401879 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393824 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393827 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393829 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393831 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393834 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393837 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393839 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393842 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393844 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393847 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393850 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393852 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393855 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393857 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393860 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393862 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393865 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393867 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393870 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393872 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:57.402357 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393875 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393878 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393880 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393883 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393885 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393887 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393892 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393894 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393897 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393899 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393902 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393904 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393906 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393909 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393911 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.393934 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:19:57.402808 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393968 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393973 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393975 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393979 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393981 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393984 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393986 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393989 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393992 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393994 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393997 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.393999 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394002 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394004 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394007 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394009 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394012 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394014 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394017 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394019 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:57.403214 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394022 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394025 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394029 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394031 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394034 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394037 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394039 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394042 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394044 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394047 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394050 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394052 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394054 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394057 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394060 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394062 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394065 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394068 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394070 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394073 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:57.403662 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394076 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394079 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394082 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394086 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394089 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394092 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394094 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394116 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394120 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394123 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394125 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394128 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394130 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394133 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394139 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394143 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394145 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394148 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:57.404163 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394151 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394153 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394156 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394158 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394161 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394163 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394166 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394168 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394171 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394173 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394176 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394178 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394181 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394183 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394185 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394188 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394191 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394193 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394196 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394198 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:57.404573 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394201 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394203 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394208 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394210 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394213 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394215 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394218 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394221 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394224 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394227 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394230 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394232 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394235 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394237 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394240 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394242 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394245 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394248 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394250 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394253 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:57.405046 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394260 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394263 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394265 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394268 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394270 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394273 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394275 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:57.394278 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.394326 2573 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.396431 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.397261 2573 server.go:1014] "Starting client certificate rotation" Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.397355 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.397388 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.399237 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:19:57.405710 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.400343 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:19:57.410760 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.410427 2573 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:19:57.413723 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.413708 2573 log.go:25] "Validated CRI v1 image API" Oct 30 19:19:57.414549 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.414522 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:19:57.417425 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.417403 2573 fs.go:135] Filesystem UUIDs: map[476531c9-3f6d-4320-a189-8ddf2182ae01:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f34170cd-8205-483d-b939-e47cefec7c36:/dev/nvme0n1p3] Oct 30 19:19:57.417486 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.417425 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:19:57.421981 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.421859 2573 manager.go:217] Machine: {Timestamp:2026-10-30 19:19:57.421380617 +0000 UTC m=+0.294567898 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3082303 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27928f3b602256eadb2787e35717d0 SystemUUID:ec27928f-3b60-2256-eadb-2787e35717d0 BootID:b3f484a7-75f2-4fea-87f6-83eac4cf4b49 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:44:90:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:44:90:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:19:63:fc:c6:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:19:57.421981 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.421983 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:19:57.422111 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.422090 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:19:57.424837 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.424813 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:19:57.424967 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.424838 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-21.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:19:57.425016 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.424969 2573 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:19:57.425016 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.424977 2573 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:19:57.425016 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.425012 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:19:57.425091 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.425029 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:19:57.426148 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.426139 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:19:57.430066 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.430054 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:19:57.430194 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.430185 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:19:57.430847 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.430839 2573 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:19:57.430883 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.430858 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:19:57.430883 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.430869 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:19:57.430883 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.430877 2573 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:19:57.430992 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.430885 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:19:57.431207 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.431195 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:19:57.433133 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.433119 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:19:57.438793 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.438780 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:19:57.439550 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.439538 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:19:57.439592 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.439569 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:19:57.439726 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.439717 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:19:57.439754 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.439734 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:19:57.439754 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.439741 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:19:57.439754 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.439748 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:19:57.439754 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.439754 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:19:57.439867 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.439762 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:19:57.440298 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.440289 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:19:57.440334 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.440300 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:19:57.440334 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.440309 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:19:57.440334 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.440315 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:19:57.440334 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.440330 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:19:57.440450 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.440342 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:19:57.440948 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.440940 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:19:57.440978 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.440951 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:19:57.444782 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.444768 2573 server.go:1263] "Started kubelet" Oct 30 19:19:57.444843 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.444804 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:19:57.444990 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.444949 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:19:57.445063 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.445004 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:19:57.446373 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.446349 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:19:57.446702 ip-10-0-138-21 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:19:57.449684 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.449663 2573 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:19:57.451056 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.451037 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:19:57.452989 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.452975 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:19:57.453581 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.453566 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:19:57.453710 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.453691 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:57.454279 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454262 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:19:57.454279 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454267 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:19:57.454429 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454290 2573 factory.go:55] Registering systemd factory Oct 30 19:19:57.454429 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454300 2573 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:19:57.454429 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454266 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:19:57.454429 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454351 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:19:57.454429 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454392 2573 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:19:57.454429 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454400 2573 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:19:57.454655 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454494 2573 factory.go:153] Registering CRI-O factory Oct 30 19:19:57.454655 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454505 2573 factory.go:223] Registration of the crio container factory successfully Oct 30 19:19:57.454655 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454521 2573 factory.go:103] Registering Raw factory Oct 30 19:19:57.454655 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.454538 2573 manager.go:1199] Started watching for new ooms in manager Oct 30 19:19:57.455391 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.455378 2573 manager.go:319] Starting recovery of all containers Oct 30 19:19:57.464778 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.464755 2573 manager.go:324] Recovery completed Oct 30 19:19:57.468954 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.468943 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:19:57.472639 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.472617 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:19:57.472715 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.472650 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:19:57.472715 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.472661 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:19:57.473151 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.473138 2573 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:19:57.473151 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.473144 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:19:57.473253 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.473149 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:19:57.473253 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.473196 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:19:57.475024 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.475007 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:19:57.475178 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.475167 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:19:57.475341 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.475322 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:19:57.476191 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.476086 2573 policy_none.go:50] "Start" Oct 30 19:19:57.476463 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.476446 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:19:57.476543 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.476474 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:19:57.476596 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.476533 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:19:57.476704 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.476450 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-21.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:19:57.477442 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.476441 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-21.ec2.internal.18e364e1c4225733 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-21.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-21.ec2.internal,},FirstTimestamp:2026-10-30 19:19:57.444593459 +0000 UTC m=+0.317780739,LastTimestamp:2026-10-30 19:19:57.444593459 +0000 UTC m=+0.317780739,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-21.ec2.internal,}" Oct 30 19:19:57.478492 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.478480 2573 policy_none.go:44] "Start" Oct 30 19:19:57.484494 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.484468 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:19:57.490019 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.489994 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:19:57.490019 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.489994 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 19:19:57.490149 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.489995 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:19:57.517035 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.516924 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6k5d8" Oct 30 19:19:57.517035 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.516995 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:19:57.521693 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.521601 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-21.ec2.internal.18e364e1c5ce43b8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-21.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-21.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-21.ec2.internal,},FirstTimestamp:2026-10-30 19:19:57.47263788 +0000 UTC m=+0.345825162,LastTimestamp:2026-10-30 19:19:57.47263788 +0000 UTC m=+0.345825162,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-21.ec2.internal,}" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.524605 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6k5d8" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.525205 2573 manager.go:341] "Starting Device Plugin manager" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.525233 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.525241 2573 server.go:87] "Starting device plugin registration server" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.525423 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.525431 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.525543 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.525631 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.525639 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.526038 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.526082 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.534481 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.535739 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.535756 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.535770 2573 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:19:57.542193 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.535801 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:19:57.550561 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.550548 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:19:57.626522 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.626476 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:19:57.627326 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.627313 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:19:57.627388 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.627344 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:19:57.627388 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.627357 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:19:57.627388 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.627378 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.636729 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.636707 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal"] Oct 30 19:19:57.636777 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.636769 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:19:57.637453 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.637439 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:19:57.637523 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.637468 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:19:57.637523 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.637478 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:19:57.639662 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.639650 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:19:57.639802 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.639789 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.639837 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.639819 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:19:57.640319 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.640301 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:19:57.640406 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.640334 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:19:57.640406 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.640304 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:19:57.640406 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.640366 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:19:57.640406 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.640378 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:19:57.640406 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.640345 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:19:57.642534 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.642519 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.642601 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.642545 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:19:57.643592 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.643577 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:19:57.643661 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.643604 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:19:57.643661 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.643612 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:19:57.644736 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.644719 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.644816 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.644740 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-21.ec2.internal\": node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:57.655945 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.655929 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62fcc3a86a753eb9d656b3c26d634e28-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" (UID: \"62fcc3a86a753eb9d656b3c26d634e28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.656020 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.655953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.656020 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.655977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.662498 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.662481 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-21.ec2.internal\" not found" node="ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.664467 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.664449 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-21.ec2.internal\" not found" node="ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.684926 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.684905 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:57.756787 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.756766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.756869 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.756798 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.756869 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.756808 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.756869 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.756830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62fcc3a86a753eb9d656b3c26d634e28-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" (UID: \"62fcc3a86a753eb9d656b3c26d634e28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.756869 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.756849 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.756994 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.756869 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62fcc3a86a753eb9d656b3c26d634e28-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" (UID: \"62fcc3a86a753eb9d656b3c26d634e28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.785717 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.785697 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:57.886149 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.886088 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:57.966273 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.966253 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.968417 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:57.968401 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:19:57.986418 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:57.986391 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:58.086917 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:58.086892 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:58.187518 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:58.187474 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:58.287914 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:58.287895 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:58.388528 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:58.388500 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:58.397683 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.397655 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:19:58.397819 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.397800 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:19:58.489256 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:58.489212 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:58.531159 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.531138 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:19:58.531276 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.531146 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:14:58 +0000 UTC" deadline="2028-04-09 07:08:18.422735672 +0000 UTC" Oct 30 19:19:58.531276 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.531185 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12635h48m19.89155529s" Oct 30 19:19:58.532425 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.532407 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:19:58.532501 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.532444 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:19:58.532501 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.532459 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:19:58.541060 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.541045 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:19:58.589689 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:58.589670 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:58.637186 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:58.637157 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b92012d9bff88a0620cb1df1375601.slice/crio-650acbdd35e837e582c04501a2cbe8dac53e5203ca5d566f3503a94a2cf1bc46 WatchSource:0}: Error finding container 650acbdd35e837e582c04501a2cbe8dac53e5203ca5d566f3503a94a2cf1bc46: Status 404 returned error can't find the container with id 650acbdd35e837e582c04501a2cbe8dac53e5203ca5d566f3503a94a2cf1bc46 Oct 30 19:19:58.639820 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:58.639798 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fcc3a86a753eb9d656b3c26d634e28.slice/crio-feb2c31699f11464ebd9050a87da5f7bf3c8d92fb4b9e9c849c6984486bb1696 WatchSource:0}: Error finding container feb2c31699f11464ebd9050a87da5f7bf3c8d92fb4b9e9c849c6984486bb1696: Status 404 returned error can't find the container with id feb2c31699f11464ebd9050a87da5f7bf3c8d92fb4b9e9c849c6984486bb1696 Oct 30 19:19:58.643199 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.643185 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:19:58.690657 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:58.690634 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:19:58.720227 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.720207 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:19:58.754317 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.754271 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:19:58.777523 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.777504 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:19:58.779166 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.779152 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:19:58.797497 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:58.797478 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:19:59.122741 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.122682 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:19:59.432360 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.432332 2573 apiserver.go:52] "Watching apiserver" Oct 30 19:19:59.453520 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.453492 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:19:59.470580 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.470555 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:19:59.470909 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.470888 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-ttpmm","kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal","openshift-dns/node-resolver-5lmmj","openshift-multus/multus-additional-cni-plugins-8sm2r","openshift-multus/network-metrics-daemon-q7pb2","openshift-network-diagnostics/network-check-target-zrrw2","openshift-cluster-node-tuning-operator/tuned-cj445","openshift-image-registry/node-ca-jsp6k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m","openshift-network-operator/iptables-alerter-czfgl","kube-system/global-pull-secret-syncer-vfmk2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal","openshift-multus/multus-h28qn","openshift-ovn-kubernetes/ovnkube-node-f7zr5"] Oct 30 19:19:59.473919 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.473899 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.476007 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.475976 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.478624 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.478600 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.480822 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.480794 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.485733 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.485716 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:19:59.485916 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.485781 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:19:59.488325 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.487991 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:19:59.488325 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.488087 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:19:59.489147 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.489052 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:19:59.489147 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.489116 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:19:59.489306 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.489160 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hxbtq\"" Oct 30 19:19:59.489651 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.489630 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gx5x8\"" Oct 30 19:19:59.489915 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.489895 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:19:59.490049 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.490031 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:19:59.490271 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.490074 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:19:59.490271 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.490128 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:19:59.490761 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.490739 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.492063 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.492047 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:19:59.492289 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.492268 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b7kzg\"" Oct 30 19:19:59.493240 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.493123 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:19:59.494466 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.494445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:19:59.497197 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.497173 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.497680 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.497657 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:19:59.497773 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.497717 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6tb2l" Oct 30 19:19:59.501078 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.501048 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.501901 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.501882 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6tb2l" Oct 30 19:19:59.505706 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.505680 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:19:59.505801 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.505754 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:19:59.508134 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.508108 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.540221 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.540149 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" event={"ID":"62fcc3a86a753eb9d656b3c26d634e28","Type":"ContainerStarted","Data":"feb2c31699f11464ebd9050a87da5f7bf3c8d92fb4b9e9c849c6984486bb1696"} Oct 30 19:19:59.542302 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.542277 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" event={"ID":"d4b92012d9bff88a0620cb1df1375601","Type":"ContainerStarted","Data":"650acbdd35e837e582c04501a2cbe8dac53e5203ca5d566f3503a94a2cf1bc46"} Oct 30 19:19:59.556155 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.556132 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:19:59.565367 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-node-log\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.565487 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565383 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0926fd0c-2705-4d4d-afbd-7007b665bb4e-iptables-alerter-script\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.565487 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-netns\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.565487 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565482 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8md4c\" (UniqueName: \"kubernetes.io/projected/f83325ae-1680-4213-94ca-12001250ffa9-kube-api-access-8md4c\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.565663 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:19:59.565663 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.565663 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565620 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6p8r\" (UniqueName: \"kubernetes.io/projected/feab0cd3-8408-4594-b808-95f5653188c2-kube-api-access-p6p8r\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.565816 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d809989-435e-4d38-807e-4bdbe414cf32-agent-certs\") pod \"konnectivity-agent-ttpmm\" (UID: \"1d809989-435e-4d38-807e-4bdbe414cf32\") " pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:19:59.565816 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-netns\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.565816 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-netd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.565969 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-socket-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.565969 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-bin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.565969 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565931 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-binary-copy\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.566121 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.565959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a94f5874-53f0-4be0-bb77-ae6d535356ef-host\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.566121 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-var-lib-kubelet\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.566121 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d809989-435e-4d38-807e-4bdbe414cf32-konnectivity-ca\") pod \"konnectivity-agent-ttpmm\" (UID: \"1d809989-435e-4d38-807e-4bdbe414cf32\") " pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:19:59.566298 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-log-socket\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.566298 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.566298 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566203 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-cnibin\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.566298 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566225 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a94f5874-53f0-4be0-bb77-ae6d535356ef-serviceca\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.566298 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysconfig\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.566298 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-lib-modules\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566308 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566336 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0926fd0c-2705-4d4d-afbd-7007b665bb4e-host-slash\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57kl\" (UniqueName: \"kubernetes.io/projected/a94f5874-53f0-4be0-bb77-ae6d535356ef-kube-api-access-j57kl\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-device-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-config\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-script-lib\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-sys\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-systemd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566489 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-env-overrides\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc66h\" (UniqueName: \"kubernetes.io/projected/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-kube-api-access-gc66h\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566540 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-cnibin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566567 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdmf4\" (UniqueName: \"kubernetes.io/projected/285534af-06ec-4171-93d8-2e3b57e1f85b-kube-api-access-xdmf4\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:19:59.566603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566593 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-slash\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovn-node-metrics-cert\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-multus\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-hostroot\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-conf-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-multus-certs\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566844 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-etc-kubernetes\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4jx\" (UniqueName: \"kubernetes.io/projected/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-kube-api-access-gr4jx\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-etc-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-bin\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.566983 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-k8s-cni-cncf-io\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-modprobe-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567106 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-ovn\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.567256 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-multus-daemon-config\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knvfn\" (UniqueName: \"kubernetes.io/projected/44b56555-103f-4cc0-8c1c-507c3cc48880-kube-api-access-knvfn\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567331 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567356 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-tmp-dir\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-kubelet-config\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-systemd-units\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-var-lib-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567468 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-system-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-system-cni-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-conf\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-systemd\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-host\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-etc-tuned\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567640 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-registration-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567664 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-sys-fs\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.567964 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-cni-binary-copy\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-socket-dir-parent\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-dbus\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567789 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-tmp\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567960 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.567990 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhgjf\" (UniqueName: \"kubernetes.io/projected/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kube-api-access-fhgjf\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.568021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-kubernetes\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.568045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-run\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.568085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbt4q\" (UniqueName: \"kubernetes.io/projected/0926fd0c-2705-4d4d-afbd-7007b665bb4e-kube-api-access-tbt4q\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.568112 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-kubelet\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.568140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-hosts-file\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.568658 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.568166 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-kubelet\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.597493 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.597472 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:19:59.597947 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.597922 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:19:59.598051 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.597960 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:19:59.598146 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598094 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9smtf\"" Oct 30 19:19:59.598146 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598132 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:19:59.598255 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598194 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:19:59.598374 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598325 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:19:59.598441 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598390 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:19:59.598441 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598422 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:19:59.598539 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598472 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gx26w\"" Oct 30 19:19:59.598539 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598500 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:19:59.598539 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598531 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:19:59.598675 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598559 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:19:59.598675 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598565 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:19:59.598675 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598624 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8rhbj\"" Oct 30 19:19:59.598675 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598645 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bkrzr\"" Oct 30 19:19:59.598675 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598505 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:19:59.598873 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598680 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:19:59.598873 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598688 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:19:59.598873 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598648 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:19:59.598873 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598731 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bcv2g\"" Oct 30 19:19:59.598873 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598775 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:19:59.598873 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598811 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:19:59.598873 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598835 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:19:59.598873 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.598857 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-w8vq8\"" Oct 30 19:19:59.668892 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.668832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-bin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.668892 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.668873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-binary-copy\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.668892 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.668892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a94f5874-53f0-4be0-bb77-ae6d535356ef-host\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.669085 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.668908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-var-lib-kubelet\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.669085 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.668968 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a94f5874-53f0-4be0-bb77-ae6d535356ef-host\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.669085 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.668968 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-bin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.669085 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.668989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d809989-435e-4d38-807e-4bdbe414cf32-konnectivity-ca\") pod \"konnectivity-agent-ttpmm\" (UID: \"1d809989-435e-4d38-807e-4bdbe414cf32\") " pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:19:59.669085 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-log-socket\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669085 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669040 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-var-lib-kubelet\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.669085 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669087 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-log-socket\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669094 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-cnibin\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a94f5874-53f0-4be0-bb77-ae6d535356ef-serviceca\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669173 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysconfig\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-lib-modules\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0926fd0c-2705-4d4d-afbd-7007b665bb4e-host-slash\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669261 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57kl\" (UniqueName: \"kubernetes.io/projected/a94f5874-53f0-4be0-bb77-ae6d535356ef-kube-api-access-j57kl\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669286 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-device-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-config\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669333 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-script-lib\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669361 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-sys\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-systemd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-env-overrides\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc66h\" (UniqueName: \"kubernetes.io/projected/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-kube-api-access-gc66h\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-cnibin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdmf4\" (UniqueName: \"kubernetes.io/projected/285534af-06ec-4171-93d8-2e3b57e1f85b-kube-api-access-xdmf4\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-slash\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovn-node-metrics-cert\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669586 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-binary-copy\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669586 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d809989-435e-4d38-807e-4bdbe414cf32-konnectivity-ca\") pod \"konnectivity-agent-ttpmm\" (UID: \"1d809989-435e-4d38-807e-4bdbe414cf32\") " pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669646 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-multus\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669683 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a94f5874-53f0-4be0-bb77-ae6d535356ef-serviceca\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-multus\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669728 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysconfig\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669768 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-systemd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.669790 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669782 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-lib-modules\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.669829 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-device-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670030 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-env-overrides\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670155 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670167 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-slash\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670209 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-config\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670232 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-sys\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670272 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-cnibin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670312 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0926fd0c-2705-4d4d-afbd-7007b665bb4e-host-slash\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-hostroot\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670355 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-cnibin\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.670380 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670379 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-conf-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670422 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-multus-certs\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670429 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670442 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670466 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-hostroot\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670491 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-script-lib\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670505 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-etc-kubernetes\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4jx\" (UniqueName: \"kubernetes.io/projected/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-kube-api-access-gr4jx\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670580 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-multus-certs\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670591 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-etc-kubernetes\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-etc-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670681 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-conf-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.670694 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670709 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-bin\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670761 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-etc-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.670961 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.670776 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs podName:285534af-06ec-4171-93d8-2e3b57e1f85b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:00.170739796 +0000 UTC m=+3.043927093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs") pod "network-metrics-daemon-q7pb2" (UID: "285534af-06ec-4171-93d8-2e3b57e1f85b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-k8s-cni-cncf-io\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670818 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-bin\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-modprobe-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670873 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-k8s-cni-cncf-io\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-ovn\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.670992 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-modprobe-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671008 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-ovn\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671008 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-multus-daemon-config\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671088 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knvfn\" (UniqueName: \"kubernetes.io/projected/44b56555-103f-4cc0-8c1c-507c3cc48880-kube-api-access-knvfn\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-tmp-dir\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-kubelet-config\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:19:59.671623 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-systemd-units\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671255 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-var-lib-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-system-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671304 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-system-cni-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671339 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-system-cni-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-conf\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-systemd\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-host\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-etc-tuned\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-registration-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-sys-fs\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671447 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-conf\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-cni-binary-copy\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-socket-dir-parent\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.672167 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-dbus\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671565 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-tmp\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671643 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhgjf\" (UniqueName: \"kubernetes.io/projected/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kube-api-access-fhgjf\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-kubernetes\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-run\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbt4q\" (UniqueName: \"kubernetes.io/projected/0926fd0c-2705-4d4d-afbd-7007b665bb4e-kube-api-access-tbt4q\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-kubelet\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-hosts-file\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-kubelet\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-node-log\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671868 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0926fd0c-2705-4d4d-afbd-7007b665bb4e-iptables-alerter-script\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-cni-binary-copy\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-netns\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-multus-daemon-config\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8md4c\" (UniqueName: \"kubernetes.io/projected/f83325ae-1680-4213-94ca-12001250ffa9-kube-api-access-8md4c\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.673001 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671955 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6p8r\" (UniqueName: \"kubernetes.io/projected/feab0cd3-8408-4594-b808-95f5653188c2-kube-api-access-p6p8r\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.671990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d809989-435e-4d38-807e-4bdbe414cf32-agent-certs\") pod \"konnectivity-agent-ttpmm\" (UID: \"1d809989-435e-4d38-807e-4bdbe414cf32\") " pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-netns\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-netd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-socket-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672094 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-kubernetes\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672132 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-run\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672174 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-socket-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672239 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-node-log\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672258 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-registration-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672331 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672477 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-sys-fs\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672523 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-netns\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672663 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-kubelet\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.673899 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672728 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-kubelet-config\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672773 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-systemd-units\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672819 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-var-lib-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672851 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-systemd\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672873 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-host\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672896 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-netd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672912 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-system-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672178 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-hosts-file\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672975 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-dbus\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.672980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-netns\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.673040 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-socket-dir-parent\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.673083 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-kubelet\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.673315 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-tmp-dir\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.673335 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.673407 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret podName:6829a2b0-e64b-4b3c-9007-966545b88c8d nodeName:}" failed. No retries permitted until 2026-10-30 19:20:00.173388267 +0000 UTC m=+3.046575545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret") pod "global-pull-secret-syncer-vfmk2" (UID: "6829a2b0-e64b-4b3c-9007-966545b88c8d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:19:59.674781 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.674124 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.678435 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.676409 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-etc-tuned\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.678435 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.676672 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-tmp\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.678937 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.678917 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0926fd0c-2705-4d4d-afbd-7007b665bb4e-iptables-alerter-script\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.679922 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.679867 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovn-node-metrics-cert\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.680026 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.679970 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.689952 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.689923 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d809989-435e-4d38-807e-4bdbe414cf32-agent-certs\") pod \"konnectivity-agent-ttpmm\" (UID: \"1d809989-435e-4d38-807e-4bdbe414cf32\") " pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:19:59.690563 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.690540 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:19:59.690656 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.690566 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:19:59.690656 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.690579 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mhc7r for pod openshift-network-diagnostics/network-check-target-zrrw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:19:59.690656 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:19:59.690643 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r podName:625ee392-6184-45a1-83a4-f577ee2062a0 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:00.190625183 +0000 UTC m=+3.063812468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mhc7r" (UniqueName: "kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r") pod "network-check-target-zrrw2" (UID: "625ee392-6184-45a1-83a4-f577ee2062a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:19:59.691628 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.691604 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57kl\" (UniqueName: \"kubernetes.io/projected/a94f5874-53f0-4be0-bb77-ae6d535356ef-kube-api-access-j57kl\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.691968 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.691946 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc66h\" (UniqueName: \"kubernetes.io/projected/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-kube-api-access-gc66h\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.693577 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.693550 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdmf4\" (UniqueName: \"kubernetes.io/projected/285534af-06ec-4171-93d8-2e3b57e1f85b-kube-api-access-xdmf4\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:19:59.694398 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.694372 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knvfn\" (UniqueName: \"kubernetes.io/projected/44b56555-103f-4cc0-8c1c-507c3cc48880-kube-api-access-knvfn\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.695374 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.694812 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhgjf\" (UniqueName: \"kubernetes.io/projected/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kube-api-access-fhgjf\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.695374 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.695318 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8md4c\" (UniqueName: \"kubernetes.io/projected/f83325ae-1680-4213-94ca-12001250ffa9-kube-api-access-8md4c\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.695963 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.695495 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbt4q\" (UniqueName: \"kubernetes.io/projected/0926fd0c-2705-4d4d-afbd-7007b665bb4e-kube-api-access-tbt4q\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.695963 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.695907 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6p8r\" (UniqueName: \"kubernetes.io/projected/feab0cd3-8408-4594-b808-95f5653188c2-kube-api-access-p6p8r\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.697603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.697584 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4jx\" (UniqueName: \"kubernetes.io/projected/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-kube-api-access-gr4jx\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.753815 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.753793 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:19:59.791475 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.791452 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:19:59.803178 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:59.803150 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeab0cd3_8408_4594_b808_95f5653188c2.slice/crio-c3f236fd2ede3ae9e3040b7b1512d9374f9828784668400e7ec3564c186cecee WatchSource:0}: Error finding container c3f236fd2ede3ae9e3040b7b1512d9374f9828784668400e7ec3564c186cecee: Status 404 returned error can't find the container with id c3f236fd2ede3ae9e3040b7b1512d9374f9828784668400e7ec3564c186cecee Oct 30 19:19:59.808062 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.808034 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h28qn" Oct 30 19:19:59.817482 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.817449 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:19:59.818785 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:59.818761 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83325ae_1680_4213_94ca_12001250ffa9.slice/crio-dda3d5b8ad4cca6cc86831b842cd8e6b948a745edda19e90592cce73f0273d6d WatchSource:0}: Error finding container dda3d5b8ad4cca6cc86831b842cd8e6b948a745edda19e90592cce73f0273d6d: Status 404 returned error can't find the container with id dda3d5b8ad4cca6cc86831b842cd8e6b948a745edda19e90592cce73f0273d6d Oct 30 19:19:59.827064 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:59.827041 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d09042e_9b40_41d8_803d_8d4a0bbf0fd1.slice/crio-137711c73ab79fd6681042fe37d0099fce4d63786c4e67a33942951554f56db0 WatchSource:0}: Error finding container 137711c73ab79fd6681042fe37d0099fce4d63786c4e67a33942951554f56db0: Status 404 returned error can't find the container with id 137711c73ab79fd6681042fe37d0099fce4d63786c4e67a33942951554f56db0 Oct 30 19:19:59.843207 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.843172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:19:59.845826 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.845756 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:19:59.847354 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.847334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:19:59.853207 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:59.853178 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b56555_103f_4cc0_8c1c_507c3cc48880.slice/crio-b9e1fc9bec15bfe68425b7bc1e9fa179cc7981afdff85b5fd2993f463f03047a WatchSource:0}: Error finding container b9e1fc9bec15bfe68425b7bc1e9fa179cc7981afdff85b5fd2993f463f03047a: Status 404 returned error can't find the container with id b9e1fc9bec15bfe68425b7bc1e9fa179cc7981afdff85b5fd2993f463f03047a Oct 30 19:19:59.857118 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:59.857090 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d809989_435e_4d38_807e_4bdbe414cf32.slice/crio-59ed34f923ef2cb34c2de5e8ab3102794367aa63950b3ecd1ae4d80937dfc0d1 WatchSource:0}: Error finding container 59ed34f923ef2cb34c2de5e8ab3102794367aa63950b3ecd1ae4d80937dfc0d1: Status 404 returned error can't find the container with id 59ed34f923ef2cb34c2de5e8ab3102794367aa63950b3ecd1ae4d80937dfc0d1 Oct 30 19:19:59.859905 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:59.859847 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a0dec1_fbc1_477b_9aaa_b921fc3ff46f.slice/crio-7abc4a27d1aa763cb4fa08cd00bb93c2cabcc3feb09667ea7daa12e6d7ae23f5 WatchSource:0}: Error finding container 7abc4a27d1aa763cb4fa08cd00bb93c2cabcc3feb09667ea7daa12e6d7ae23f5: Status 404 returned error can't find the container with id 7abc4a27d1aa763cb4fa08cd00bb93c2cabcc3feb09667ea7daa12e6d7ae23f5 Oct 30 19:19:59.866135 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.866095 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:19:59.868234 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.868215 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:19:59.871443 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:19:59.871421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:19:59.877633 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:59.877607 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0926fd0c_2705_4d4d_afbd_7007b665bb4e.slice/crio-ffa802b816fc4ebcd018f4b21d01c2acab4fa247c028c3bf4e8051dcc08a0426 WatchSource:0}: Error finding container ffa802b816fc4ebcd018f4b21d01c2acab4fa247c028c3bf4e8051dcc08a0426: Status 404 returned error can't find the container with id ffa802b816fc4ebcd018f4b21d01c2acab4fa247c028c3bf4e8051dcc08a0426 Oct 30 19:19:59.884451 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:19:59.884430 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a4eb6e_f33f_4f60_9193_2dfbb3c2003e.slice/crio-f7ccea346eba8ce24bc800c05a1d30d659886a5e029fcfb289944af885ae6a96 WatchSource:0}: Error finding container f7ccea346eba8ce24bc800c05a1d30d659886a5e029fcfb289944af885ae6a96: Status 404 returned error can't find the container with id f7ccea346eba8ce24bc800c05a1d30d659886a5e029fcfb289944af885ae6a96 Oct 30 19:20:00.174976 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.174943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:00.175096 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.175015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:00.175281 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:00.175261 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:20:00.175372 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:00.175334 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs podName:285534af-06ec-4171-93d8-2e3b57e1f85b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:01.175308271 +0000 UTC m=+4.048495541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs") pod "network-metrics-daemon-q7pb2" (UID: "285534af-06ec-4171-93d8-2e3b57e1f85b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:20:00.175863 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:00.175824 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:20:00.175940 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:00.175905 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret podName:6829a2b0-e64b-4b3c-9007-966545b88c8d nodeName:}" failed. No retries permitted until 2026-10-30 19:20:01.175890029 +0000 UTC m=+4.049077302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret") pod "global-pull-secret-syncer-vfmk2" (UID: "6829a2b0-e64b-4b3c-9007-966545b88c8d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:20:00.276010 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.275846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:00.276626 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:00.276212 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:20:00.276626 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:00.276238 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:20:00.276626 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:00.276252 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mhc7r for pod openshift-network-diagnostics/network-check-target-zrrw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:20:00.276626 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:00.276307 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r podName:625ee392-6184-45a1-83a4-f577ee2062a0 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:01.276289649 +0000 UTC m=+4.149476917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhc7r" (UniqueName: "kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r") pod "network-check-target-zrrw2" (UID: "625ee392-6184-45a1-83a4-f577ee2062a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:20:00.508619 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.508517 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:15:00 +0000 UTC" deadline="2028-05-23 01:34:39.289620294 +0000 UTC" Oct 30 19:20:00.508619 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.508583 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13686h14m38.781041761s" Oct 30 19:20:00.551246 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.551120 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5lmmj" event={"ID":"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1","Type":"ContainerStarted","Data":"137711c73ab79fd6681042fe37d0099fce4d63786c4e67a33942951554f56db0"} Oct 30 19:20:00.553185 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.553081 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h28qn" event={"ID":"f83325ae-1680-4213-94ca-12001250ffa9","Type":"ContainerStarted","Data":"dda3d5b8ad4cca6cc86831b842cd8e6b948a745edda19e90592cce73f0273d6d"} Oct 30 19:20:00.560105 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.559445 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" event={"ID":"62fcc3a86a753eb9d656b3c26d634e28","Type":"ContainerStarted","Data":"f29b70b4d2cbb236b2ebd3aeb3901a34e4c7ac48700a4239a66c9a377e44a9c3"} Oct 30 19:20:00.573000 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.572837 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" event={"ID":"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e","Type":"ContainerStarted","Data":"f7ccea346eba8ce24bc800c05a1d30d659886a5e029fcfb289944af885ae6a96"} Oct 30 19:20:00.577353 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.577330 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cj445" event={"ID":"feab0cd3-8408-4594-b808-95f5653188c2","Type":"ContainerStarted","Data":"c3f236fd2ede3ae9e3040b7b1512d9374f9828784668400e7ec3564c186cecee"} Oct 30 19:20:00.585390 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.585305 2573 generic.go:356] "Generic (PLEG): container finished" podID="d4b92012d9bff88a0620cb1df1375601" containerID="365620a125dc1e8c6d51303685712212ef9ebae9055e3b253efa48772ac550b4" exitCode=0 Oct 30 19:20:00.585390 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.585341 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" event={"ID":"d4b92012d9bff88a0620cb1df1375601","Type":"ContainerDied","Data":"365620a125dc1e8c6d51303685712212ef9ebae9055e3b253efa48772ac550b4"} Oct 30 19:20:00.612955 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.610246 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jsp6k" event={"ID":"a94f5874-53f0-4be0-bb77-ae6d535356ef","Type":"ContainerStarted","Data":"81cb7bada7e07c00a9ef3c3d469461a4e77d2336142fd49476735e211ba35710"} Oct 30 19:20:00.614401 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.614354 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-czfgl" event={"ID":"0926fd0c-2705-4d4d-afbd-7007b665bb4e","Type":"ContainerStarted","Data":"ffa802b816fc4ebcd018f4b21d01c2acab4fa247c028c3bf4e8051dcc08a0426"} Oct 30 19:20:00.626678 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.626652 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" event={"ID":"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f","Type":"ContainerStarted","Data":"7abc4a27d1aa763cb4fa08cd00bb93c2cabcc3feb09667ea7daa12e6d7ae23f5"} Oct 30 19:20:00.638865 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.638807 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" podStartSLOduration=1.638790339 podStartE2EDuration="1.638790339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:00.598303976 +0000 UTC m=+3.471491279" watchObservedRunningTime="2026-10-30 19:20:00.638790339 +0000 UTC m=+3.511977637" Oct 30 19:20:00.646559 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.646396 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ttpmm" event={"ID":"1d809989-435e-4d38-807e-4bdbe414cf32","Type":"ContainerStarted","Data":"59ed34f923ef2cb34c2de5e8ab3102794367aa63950b3ecd1ae4d80937dfc0d1"} Oct 30 19:20:00.655217 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:00.655179 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" event={"ID":"44b56555-103f-4cc0-8c1c-507c3cc48880","Type":"ContainerStarted","Data":"b9e1fc9bec15bfe68425b7bc1e9fa179cc7981afdff85b5fd2993f463f03047a"} Oct 30 19:20:01.184680 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:01.184597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:01.184680 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:01.184667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:01.184902 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.184817 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:20:01.184902 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.184876 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs podName:285534af-06ec-4171-93d8-2e3b57e1f85b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:03.184856685 +0000 UTC m=+6.058043971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs") pod "network-metrics-daemon-q7pb2" (UID: "285534af-06ec-4171-93d8-2e3b57e1f85b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:20:01.185022 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.184943 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:20:01.185022 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.184977 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret podName:6829a2b0-e64b-4b3c-9007-966545b88c8d nodeName:}" failed. No retries permitted until 2026-10-30 19:20:03.184965469 +0000 UTC m=+6.058152759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret") pod "global-pull-secret-syncer-vfmk2" (UID: "6829a2b0-e64b-4b3c-9007-966545b88c8d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:20:01.286043 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:01.286007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:01.286245 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.286178 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:20:01.286245 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.286202 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:20:01.286245 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.286216 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mhc7r for pod openshift-network-diagnostics/network-check-target-zrrw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:20:01.286413 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.286273 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r podName:625ee392-6184-45a1-83a4-f577ee2062a0 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:03.28625443 +0000 UTC m=+6.159441705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhc7r" (UniqueName: "kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r") pod "network-check-target-zrrw2" (UID: "625ee392-6184-45a1-83a4-f577ee2062a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:20:01.509251 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:01.509160 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:15:00 +0000 UTC" deadline="2028-07-20 02:02:45.073748704 +0000 UTC" Oct 30 19:20:01.509251 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:01.509202 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15078h42m43.564551792s" Oct 30 19:20:01.538774 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:01.538753 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:01.538903 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.538862 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:01.538978 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:01.538931 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:01.539032 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.539003 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:01.539087 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:01.539061 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:01.539201 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:01.539134 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:01.667059 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:01.666993 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" event={"ID":"d4b92012d9bff88a0620cb1df1375601","Type":"ContainerStarted","Data":"dea43ea506a7ac54ce6d954d86880737a0e5dccac675835bd8c89c72de01275e"} Oct 30 19:20:02.680390 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:02.680283 2573 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="2c08f0884dc3cf6836b897c65d3d52797e9fb10eb995952e0121fa554689f80a" exitCode=0 Oct 30 19:20:02.681387 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:02.681358 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" event={"ID":"44b56555-103f-4cc0-8c1c-507c3cc48880","Type":"ContainerDied","Data":"2c08f0884dc3cf6836b897c65d3d52797e9fb10eb995952e0121fa554689f80a"} Oct 30 19:20:02.748145 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:02.748086 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" podStartSLOduration=3.748066974 podStartE2EDuration="3.748066974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:01.70682408 +0000 UTC m=+4.580011373" watchObservedRunningTime="2026-10-30 19:20:02.748066974 +0000 UTC m=+5.621254268" Oct 30 19:20:03.203046 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:03.203009 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:03.203247 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:03.203098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:03.203247 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.203242 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:20:03.203377 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.203317 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret podName:6829a2b0-e64b-4b3c-9007-966545b88c8d nodeName:}" failed. No retries permitted until 2026-10-30 19:20:07.203296581 +0000 UTC m=+10.076483852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret") pod "global-pull-secret-syncer-vfmk2" (UID: "6829a2b0-e64b-4b3c-9007-966545b88c8d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:20:03.203789 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.203767 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:20:03.203894 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.203824 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs podName:285534af-06ec-4171-93d8-2e3b57e1f85b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:07.203808863 +0000 UTC m=+10.076996139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs") pod "network-metrics-daemon-q7pb2" (UID: "285534af-06ec-4171-93d8-2e3b57e1f85b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:20:03.303518 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:03.303477 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:03.312576 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.312526 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:20:03.312685 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.312584 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:20:03.312685 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.312608 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mhc7r for pod openshift-network-diagnostics/network-check-target-zrrw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:20:03.312685 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.312674 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r podName:625ee392-6184-45a1-83a4-f577ee2062a0 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:07.312646877 +0000 UTC m=+10.185834154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhc7r" (UniqueName: "kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r") pod "network-check-target-zrrw2" (UID: "625ee392-6184-45a1-83a4-f577ee2062a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:20:03.540554 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:03.540478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:03.540709 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.540611 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:03.541035 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:03.541012 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:03.541203 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.541164 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:03.541290 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:03.541275 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:03.541377 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:03.541359 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:04.691261 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:04.691220 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ttpmm" event={"ID":"1d809989-435e-4d38-807e-4bdbe414cf32","Type":"ContainerStarted","Data":"a2f86019dc8d254d0973c962386a3a843d7a2c64bb8a6bc181342abd559090f4"} Oct 30 19:20:04.967294 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:04.967168 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:20:04.968254 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:04.967979 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:20:05.005507 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:05.005453 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ttpmm" podStartSLOduration=3.245610903 podStartE2EDuration="7.005435319s" totalImagesPullingTime="3.759824416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:04.729600088 +0000 UTC m=+7.602787383" watchObservedRunningTime="2026-10-30 19:20:05.005435319 +0000 UTC m=+7.878622614" Oct 30 19:20:05.538830 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:05.538803 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:05.538976 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:05.538929 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:05.539334 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:05.539310 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:05.539439 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:05.539422 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:05.539513 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:05.539472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:05.539571 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:05.539543 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:05.702002 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:05.701920 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jsp6k" event={"ID":"a94f5874-53f0-4be0-bb77-ae6d535356ef","Type":"ContainerStarted","Data":"0f453abb23bab5ccc20f776ef2ae18fd7e5823875e9f581e9ff70c0b02259141"} Oct 30 19:20:06.709991 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:06.709953 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" event={"ID":"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f","Type":"ContainerStarted","Data":"0825f1f2ee4f8e0efa1af97c6235ae31233617b5020a7bda0b744b97203daa8f"} Oct 30 19:20:06.710517 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:06.710065 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:20:07.234562 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:07.234531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:07.234761 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:07.234583 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:07.234761 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.234716 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:20:07.234761 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.234740 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:20:07.234958 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.234787 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs podName:285534af-06ec-4171-93d8-2e3b57e1f85b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:15.23477476 +0000 UTC m=+18.107962044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs") pod "network-metrics-daemon-q7pb2" (UID: "285534af-06ec-4171-93d8-2e3b57e1f85b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:20:07.234958 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.234800 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret podName:6829a2b0-e64b-4b3c-9007-966545b88c8d nodeName:}" failed. No retries permitted until 2026-10-30 19:20:15.234793852 +0000 UTC m=+18.107981121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret") pod "global-pull-secret-syncer-vfmk2" (UID: "6829a2b0-e64b-4b3c-9007-966545b88c8d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:20:07.335000 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:07.334956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:07.335237 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.335202 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:20:07.335317 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.335251 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:20:07.335317 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.335267 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mhc7r for pod openshift-network-diagnostics/network-check-target-zrrw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:20:07.335424 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.335318 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r podName:625ee392-6184-45a1-83a4-f577ee2062a0 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:15.335304911 +0000 UTC m=+18.208492195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhc7r" (UniqueName: "kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r") pod "network-check-target-zrrw2" (UID: "625ee392-6184-45a1-83a4-f577ee2062a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:20:07.541223 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:07.539612 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:07.541223 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.539755 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:07.541223 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:07.540209 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:07.541223 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.540308 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:07.541425 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:07.541398 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:07.541556 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:07.541500 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:07.720288 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:07.720222 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h28qn" event={"ID":"f83325ae-1680-4213-94ca-12001250ffa9","Type":"ContainerStarted","Data":"af5a6c68239cdc45865666e5e13797fea65eccce47b756a1571cff27e16c60a2"} Oct 30 19:20:07.732002 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:07.731961 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:20:07.767264 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:07.766615 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jsp6k" podStartSLOduration=4.338524006 podStartE2EDuration="9.766594283s" totalImagesPullingTime="5.428070277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:05.744114482 +0000 UTC m=+8.617301777" watchObservedRunningTime="2026-10-30 19:20:07.766594283 +0000 UTC m=+10.639781579" Oct 30 19:20:08.535215 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:08.534999 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:20:07.731982639Z","UUID":"41a3921c-dc6a-4134-b40a-9a7725688561","Handler":null,"Name":"","Endpoint":""} Oct 30 19:20:08.537296 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:08.537262 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:20:08.537296 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:08.537293 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:20:08.728366 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:08.728329 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" event={"ID":"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f","Type":"ContainerStarted","Data":"1aceb9bbe487b5ca4544f72b0b002f4db839b16b035ad69314a17e06ea32a023"} Oct 30 19:20:08.731266 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:08.731219 2573 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="bef4fc9838e8c303159651e130642aca0378aeb232bde159048faf6668305de6" exitCode=0 Oct 30 19:20:08.731374 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:08.731289 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" event={"ID":"44b56555-103f-4cc0-8c1c-507c3cc48880","Type":"ContainerDied","Data":"bef4fc9838e8c303159651e130642aca0378aeb232bde159048faf6668305de6"} Oct 30 19:20:08.734673 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:08.734649 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5lmmj" event={"ID":"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1","Type":"ContainerStarted","Data":"803548a04bed9cb3210eaf212243998656719c06abb9887ec60fdd25f30659c2"} Oct 30 19:20:08.783428 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:08.783340 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-h28qn" podStartSLOduration=3.13210057 podStartE2EDuration="10.783326726s" totalImagesPullingTime="7.651226156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:07.767356961 +0000 UTC m=+10.640544257" watchObservedRunningTime="2026-10-30 19:20:08.783326726 +0000 UTC m=+11.656514019" Oct 30 19:20:09.538104 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:09.537265 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:09.538104 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:09.537603 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:09.538104 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:09.537909 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:09.538104 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:09.537988 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:09.538104 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:09.538039 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:09.538104 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:09.538082 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:09.739169 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:09.739071 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" event={"ID":"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f","Type":"ContainerStarted","Data":"5fe73747222cd5a8f6c26bdcec017d8297cf59a7d86f3509f029031becd43b4d"} Oct 30 19:20:09.741987 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:09.741952 2573 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="c432066bc58a6c13a01a7870fc0208979114837b0fc498ffe10aa3cb5b7477f2" exitCode=0 Oct 30 19:20:09.742493 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:09.742467 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" event={"ID":"44b56555-103f-4cc0-8c1c-507c3cc48880","Type":"ContainerDied","Data":"c432066bc58a6c13a01a7870fc0208979114837b0fc498ffe10aa3cb5b7477f2"} Oct 30 19:20:09.779300 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:09.779247 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" podStartSLOduration=3.251632512 podStartE2EDuration="11.779229959s" totalImagesPullingTime="8.527597447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:09.779035111 +0000 UTC m=+12.652222407" watchObservedRunningTime="2026-10-30 19:20:09.779229959 +0000 UTC m=+12.652417253" Oct 30 19:20:09.779784 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:09.779755 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5lmmj" podStartSLOduration=3.105135185 podStartE2EDuration="11.779745855s" totalImagesPullingTime="8.67461067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:08.824835804 +0000 UTC m=+11.698023128" watchObservedRunningTime="2026-10-30 19:20:09.779745855 +0000 UTC m=+12.652933145" Oct 30 19:20:10.747438 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:10.747405 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cj445" event={"ID":"feab0cd3-8408-4594-b808-95f5653188c2","Type":"ContainerStarted","Data":"3fcabc981f0617eb8b5ee69a049aa7cdb777d4660430351ff7d5012249ea85f8"} Oct 30 19:20:10.750367 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:10.750337 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-czfgl" event={"ID":"0926fd0c-2705-4d4d-afbd-7007b665bb4e","Type":"ContainerStarted","Data":"b6a5cc66e78c2fe8b67cd7764a2365930dbfe350a596d470f82c74ae78b07b22"} Oct 30 19:20:10.789464 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:10.789409 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cj445" podStartSLOduration=1.95051091 podStartE2EDuration="12.789390497s" totalImagesPullingTime="10.838879587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:10.788868964 +0000 UTC m=+13.662056256" watchObservedRunningTime="2026-10-30 19:20:10.789390497 +0000 UTC m=+13.662577791" Oct 30 19:20:10.827329 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:10.827273 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-czfgl" podStartSLOduration=4.5292481129999995 podStartE2EDuration="12.82725405s" totalImagesPullingTime="8.298005937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:10.826699104 +0000 UTC m=+13.699886398" watchObservedRunningTime="2026-10-30 19:20:10.82725405 +0000 UTC m=+13.700441345" Oct 30 19:20:11.536406 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:11.536376 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:11.536567 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:11.536455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:11.536567 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:11.536467 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:11.536648 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:11.536574 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:11.536685 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:11.536637 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:11.536716 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:11.536700 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:11.754998 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:11.754963 2573 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="0f03e039dd550729ef3dc994e58c8dd020973ececd24afe02458f0885767193f" exitCode=0 Oct 30 19:20:11.755697 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:11.755054 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" event={"ID":"44b56555-103f-4cc0-8c1c-507c3cc48880","Type":"ContainerDied","Data":"0f03e039dd550729ef3dc994e58c8dd020973ececd24afe02458f0885767193f"} Oct 30 19:20:13.536413 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:13.536384 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:13.537169 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:13.536384 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:13.537169 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:13.536500 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:13.537169 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:13.536600 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:13.537169 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:13.536388 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:13.537169 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:13.536720 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:13.762464 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:13.762227 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" event={"ID":"44b56555-103f-4cc0-8c1c-507c3cc48880","Type":"ContainerStarted","Data":"fda8c7f84c64ad867cd5ff2f000f7ef2b7ccd8e7313c2d889fd1ea7235128f08"} Oct 30 19:20:14.719534 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:14.719502 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:20:14.720006 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:14.719662 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:20:14.720006 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:14.719972 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:20:14.766190 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:14.766165 2573 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="fda8c7f84c64ad867cd5ff2f000f7ef2b7ccd8e7313c2d889fd1ea7235128f08" exitCode=0 Oct 30 19:20:14.766311 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:14.766240 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" event={"ID":"44b56555-103f-4cc0-8c1c-507c3cc48880","Type":"ContainerDied","Data":"fda8c7f84c64ad867cd5ff2f000f7ef2b7ccd8e7313c2d889fd1ea7235128f08"} Oct 30 19:20:15.299769 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.299746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:15.299870 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.299788 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:15.299930 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.299868 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:20:15.299930 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.299893 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:20:15.299930 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.299908 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret podName:6829a2b0-e64b-4b3c-9007-966545b88c8d nodeName:}" failed. No retries permitted until 2026-10-30 19:20:31.299897895 +0000 UTC m=+34.173085164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret") pod "global-pull-secret-syncer-vfmk2" (UID: "6829a2b0-e64b-4b3c-9007-966545b88c8d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:20:15.300036 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.299948 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs podName:285534af-06ec-4171-93d8-2e3b57e1f85b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:31.299931965 +0000 UTC m=+34.173119247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs") pod "network-metrics-daemon-q7pb2" (UID: "285534af-06ec-4171-93d8-2e3b57e1f85b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:20:15.400699 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.400678 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:15.400799 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.400788 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:20:15.400835 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.400804 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:20:15.400835 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.400813 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mhc7r for pod openshift-network-diagnostics/network-check-target-zrrw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:20:15.400905 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.400849 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r podName:625ee392-6184-45a1-83a4-f577ee2062a0 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:31.400838983 +0000 UTC m=+34.274026252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhc7r" (UniqueName: "kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r") pod "network-check-target-zrrw2" (UID: "625ee392-6184-45a1-83a4-f577ee2062a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:20:15.536392 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.536371 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:15.536538 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.536517 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:15.536586 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.536543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:15.536639 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.536521 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:15.536639 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.536622 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:15.536724 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:15.536683 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:15.770976 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.770755 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" event={"ID":"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e","Type":"ContainerStarted","Data":"2fd21d0ff7633ef4b2b1a57abcdc10d2a5b2e98964ae8f6f2f73c6351fc2aa59"} Oct 30 19:20:15.771349 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.770991 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" event={"ID":"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e","Type":"ContainerStarted","Data":"33c2a79e85829ebba29cf6a9ab15428bc370bb44d9962813ef98903f510c2c2a"} Oct 30 19:20:15.771349 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.771039 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" event={"ID":"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e","Type":"ContainerStarted","Data":"a78809e414d694f4473d746d5fbea507df51a5e071a83c1371c9c60c5abaecf0"} Oct 30 19:20:15.771349 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.771082 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" event={"ID":"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e","Type":"ContainerStarted","Data":"9e1ca251ba0f1419c90f80aa9d127dce95bc586818efb1b15719e3306d628cdf"} Oct 30 19:20:15.771349 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.771139 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" event={"ID":"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e","Type":"ContainerStarted","Data":"58f950038492f646fb7cdf7fef90a2fa20bc9f1d7a85244ebc3f57c37c00d3c7"} Oct 30 19:20:15.771349 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.771177 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" event={"ID":"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e","Type":"ContainerStarted","Data":"9b27098a7e88bd9828b836e791b6b76d46fc6db90f98db07a020da2a3b430d3d"} Oct 30 19:20:15.773818 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.773792 2573 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="348c838916d935d45e7d091be2e34a7df385ca4ab42816188cb56f227b749585" exitCode=0 Oct 30 19:20:15.773923 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:15.773832 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" event={"ID":"44b56555-103f-4cc0-8c1c-507c3cc48880","Type":"ContainerDied","Data":"348c838916d935d45e7d091be2e34a7df385ca4ab42816188cb56f227b749585"} Oct 30 19:20:16.779883 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:16.779838 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" event={"ID":"44b56555-103f-4cc0-8c1c-507c3cc48880","Type":"ContainerStarted","Data":"9a3a820220ff88958dd8611129da06efc3dcb123e58a93fe11bc74cf7e525438"} Oct 30 19:20:16.840474 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:16.840417 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8sm2r" podStartSLOduration=8.038611939 podStartE2EDuration="18.840403727s" totalImagesPullingTime="10.801791788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:16.829456431 +0000 UTC m=+19.702643721" watchObservedRunningTime="2026-10-30 19:20:16.840403727 +0000 UTC m=+19.713591016" Oct 30 19:20:17.536950 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:17.536921 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:17.537082 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:17.537008 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:17.537082 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:17.537057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:17.537191 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:17.537131 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:17.537234 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:17.537194 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:17.537311 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:17.537291 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:17.785602 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:17.785571 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" event={"ID":"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e","Type":"ContainerStarted","Data":"8465b3cf80d9d2dbf20aa04da82d1b0adb71294364c0aabca253e43a9176de46"} Oct 30 19:20:19.536371 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:19.536330 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:19.536371 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:19.536355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:19.536878 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:19.536346 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:19.536878 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:19.536473 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:19.536878 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:19.536577 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:19.536878 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:19.536675 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:19.793290 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:19.793207 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" event={"ID":"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e","Type":"ContainerStarted","Data":"94d8c4482b083c3438a9d4f608a4daa5c16bf135549a883e93b13b4e7953b559"} Oct 30 19:20:19.793481 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:19.793461 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:19.793575 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:19.793539 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:19.809034 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:19.809003 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:19.809474 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:19.809457 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:19.874138 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:19.874077 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" podStartSLOduration=6.500372752 podStartE2EDuration="21.87406563s" totalImagesPullingTime="15.373692878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:19.873754199 +0000 UTC m=+22.746941489" watchObservedRunningTime="2026-10-30 19:20:19.87406563 +0000 UTC m=+22.747252920" Oct 30 19:20:20.795563 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:20.795538 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:20:21.536479 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:21.536455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:21.536608 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:21.536454 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:21.536690 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:21.536455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:21.536758 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:21.536682 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:21.536758 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:21.536553 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:21.536758 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:21.536718 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:21.797629 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:21.797572 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:20:21.831770 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:21.831745 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zrrw2"] Oct 30 19:20:21.831888 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:21.831862 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:21.831974 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:21.831957 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:21.835775 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:21.835752 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vfmk2"] Oct 30 19:20:21.835909 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:21.835857 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:21.835972 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:21.835951 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:21.842347 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:21.842326 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q7pb2"] Oct 30 19:20:21.842453 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:21.842419 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:21.842517 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:21.842501 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:23.536411 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:23.536380 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:23.536411 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:23.536391 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:23.536411 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:23.536407 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:23.536924 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:23.536505 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:23.536924 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:23.536576 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:23.536924 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:23.536671 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:25.536486 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:25.536273 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:25.536931 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:25.536311 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:25.536931 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:25.536561 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:25.536931 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:25.536708 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:25.536931 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:25.536370 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:25.536931 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:25.536843 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:27.536997 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:27.536967 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:27.537521 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:27.537050 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7pb2" podUID="285534af-06ec-4171-93d8-2e3b57e1f85b" Oct 30 19:20:27.537521 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:27.537176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:27.537521 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:27.537282 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrrw2" podUID="625ee392-6184-45a1-83a4-f577ee2062a0" Oct 30 19:20:27.537521 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:27.537346 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:27.537521 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:27.537427 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vfmk2" podUID="6829a2b0-e64b-4b3c-9007-966545b88c8d" Oct 30 19:20:27.849422 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:27.849363 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:27.849562 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:27.849549 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:20:27.860346 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:27.860325 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:28.294853 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.294801 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeReady" Oct 30 19:20:28.294940 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.294893 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:20:28.338584 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.338563 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qgdzh"] Oct 30 19:20:28.361830 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.361784 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c6bdc"] Oct 30 19:20:28.362050 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.362003 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qgdzh" Oct 30 19:20:28.366647 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.366613 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gt86s"] Oct 30 19:20:28.367732 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.367709 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c6bdc" Oct 30 19:20:28.381558 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.380738 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.381558 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.381315 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.384451 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.384428 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d984dbd48-qljg6"] Oct 30 19:20:28.384550 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.384498 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gt86s" Oct 30 19:20:28.400434 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.400413 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q"] Oct 30 19:20:28.400526 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.400515 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.414893 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.414873 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:20:28.415091 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.415077 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:20:28.415173 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.415162 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vp4s5\"" Oct 30 19:20:28.415285 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.415272 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:20:28.424423 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.424407 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9g8dh"] Oct 30 19:20:28.424542 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.424528 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:28.438637 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.438620 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56bc986487-ljfn2"] Oct 30 19:20:28.438773 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.438756 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.458381 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.458364 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cf497f4fc-s2vpv"] Oct 30 19:20:28.458474 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.458461 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.474532 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.474511 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-58fgj"] Oct 30 19:20:28.474625 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.474606 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.482526 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.482511 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.482526 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.482523 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.482697 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.482682 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.482753 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.482715 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.482753 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.482740 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 19:20:28.482987 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.482972 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 19:20:28.482987 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.482983 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.483347 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.483333 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.488784 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.488769 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt"] Oct 30 19:20:28.488863 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.488853 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.493445 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.493418 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 19:20:28.494801 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.494784 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:20:28.494873 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.494823 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 19:20:28.501793 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.501777 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.501880 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.501865 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:20:28.502094 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.502079 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.504032 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-config\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.504126 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:28.504183 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-installation-pull-secrets\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.504222 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504191 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56b5\" (UniqueName: \"kubernetes.io/projected/9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1-kube-api-access-t56b5\") pod \"kas-connection-checker-685895c7c9-qgdzh\" (UID: \"9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1\") " pod="kube-system/kas-connection-checker-685895c7c9-qgdzh" Oct 30 19:20:28.504222 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504209 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-kube-api-access-s5qrf\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.504318 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504224 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44dqf\" (UniqueName: \"kubernetes.io/projected/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-kube-api-access-44dqf\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.504318 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpf8n\" (UniqueName: \"kubernetes.io/projected/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-kube-api-access-vpf8n\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:28.504423 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-image-registry-private-configuration\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.504474 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504419 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.504474 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-bound-sa-token\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.504586 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdqlw\" (UniqueName: \"kubernetes.io/projected/b07766d9-9884-4f58-9c94-e59540953a06-kube-api-access-pdqlw\") pod \"kas-connection-checker-685895c7c9-c6bdc\" (UID: \"b07766d9-9884-4f58-9c94-e59540953a06\") " pod="kube-system/kas-connection-checker-685895c7c9-c6bdc" Oct 30 19:20:28.504586 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4cf2efa-495e-452f-b0a6-b17c1d54038b-ca-trust-extracted\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.504586 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-trusted-ca\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.504744 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504608 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ghlh\" (UniqueName: \"kubernetes.io/projected/e6811b53-7458-4b8c-b4f9-9f46fb029eea-kube-api-access-9ghlh\") pod \"kas-connection-checker-685895c7c9-gt86s\" (UID: \"e6811b53-7458-4b8c-b4f9-9f46fb029eea\") " pod="kube-system/kas-connection-checker-685895c7c9-gt86s" Oct 30 19:20:28.504744 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-trusted-ca\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.504744 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:28.504744 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504736 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-certificates\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.504956 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.504754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-serving-cert\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.505115 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.505080 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:20:28.511236 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.511221 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42"] Oct 30 19:20:28.511330 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.511318 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt" Oct 30 19:20:28.515282 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.515263 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 19:20:28.515374 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.515358 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 19:20:28.515430 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.515376 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 19:20:28.515550 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.515534 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 19:20:28.515650 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.515609 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4v55m\"" Oct 30 19:20:28.515650 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.515616 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hh67w\"" Oct 30 19:20:28.515650 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.515550 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j2d4c\"" Oct 30 19:20:28.515650 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.515580 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 19:20:28.515650 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.515551 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 19:20:28.515816 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.515553 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k56vl\"" Oct 30 19:20:28.535336 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.535313 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk"] Oct 30 19:20:28.535476 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.535462 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.559048 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.559004 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8ns8s"] Oct 30 19:20:28.559362 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.559130 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.573373 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.573358 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx"] Oct 30 19:20:28.573462 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.573450 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:28.582555 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.582504 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 19:20:28.582555 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.582514 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.582555 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.582552 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.582750 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.582595 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.582750 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.582622 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.582750 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.582628 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.582750 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.582661 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.582750 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.582670 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 19:20:28.586077 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.586061 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 19:20:28.593263 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.593248 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt"] Oct 30 19:20:28.593405 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.593390 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:28.605794 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.605776 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.605870 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.605803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpf8n\" (UniqueName: \"kubernetes.io/projected/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-kube-api-access-vpf8n\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:28.605870 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.605818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1697d46a-5b9e-429a-986f-76e472a662a5-ca-trust-extracted\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.605870 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.605833 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-registry-certificates\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.605870 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.605856 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-image-registry-private-configuration\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.605991 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.605889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-bound-sa-token\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.605991 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.605929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqlw\" (UniqueName: \"kubernetes.io/projected/b07766d9-9884-4f58-9c94-e59540953a06-kube-api-access-pdqlw\") pod \"kas-connection-checker-685895c7c9-c6bdc\" (UID: \"b07766d9-9884-4f58-9c94-e59540953a06\") " pod="kube-system/kas-connection-checker-685895c7c9-c6bdc" Oct 30 19:20:28.606053 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-installation-pull-secrets\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.606085 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4cf2efa-495e-452f-b0a6-b17c1d54038b-ca-trust-extracted\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.606137 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-tmp\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.606185 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e8615b0-abab-426b-a0ee-893109807207-serving-cert\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.606221 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606201 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnlzn\" (UniqueName: \"kubernetes.io/projected/f8e59f23-4988-41f7-b462-912ac6c7c125-kube-api-access-fnlzn\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.606267 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-bound-sa-token\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.606308 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-certificates\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.606308 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-config\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.606370 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-service-ca-bundle\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.606414 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606401 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4cf2efa-495e-452f-b0a6-b17c1d54038b-ca-trust-extracted\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.606464 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-snapshots\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.606517 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-image-registry-private-configuration\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.606573 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606516 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lkf8\" (UniqueName: \"kubernetes.io/projected/605b308f-b6a9-427f-a365-c0f8b110da64-kube-api-access-2lkf8\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.606573 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-default-certificate\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.606671 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.606727 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.606678 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:28.606727 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.606691 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d984dbd48-qljg6: secret "image-registry-tls" not found Oct 30 19:20:28.606821 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.606735 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls podName:a4cf2efa-495e-452f-b0a6-b17c1d54038b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:29.106724202 +0000 UTC m=+31.979911470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls") pod "image-registry-6d984dbd48-qljg6" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b") : secret "image-registry-tls" not found Oct 30 19:20:28.606821 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm957\" (UniqueName: \"kubernetes.io/projected/2e8615b0-abab-426b-a0ee-893109807207-kube-api-access-wm957\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.606821 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606783 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8e59f23-4988-41f7-b462-912ac6c7c125-config\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.606821 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-trusted-ca\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.606821 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606807 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-certificates\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.607125 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghlh\" (UniqueName: \"kubernetes.io/projected/e6811b53-7458-4b8c-b4f9-9f46fb029eea-kube-api-access-9ghlh\") pod \"kas-connection-checker-685895c7c9-gt86s\" (UID: \"e6811b53-7458-4b8c-b4f9-9f46fb029eea\") " pod="kube-system/kas-connection-checker-685895c7c9-gt86s" Oct 30 19:20:28.607125 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.606978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:28.607125 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-trusted-ca\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.607125 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-serving-cert\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.607125 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607040 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.607125 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607060 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-config\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.607125 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkqc\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-kube-api-access-lvkqc\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.607464 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.607464 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.607464 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.607179 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:28.607464 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607201 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqt56\" (UniqueName: \"kubernetes.io/projected/c2f5e9c8-28aa-4956-a3ee-811d5577ec5d-kube-api-access-zqt56\") pod \"volume-data-source-validator-58c9f5f46d-fj5bt\" (UID: \"c2f5e9c8-28aa-4956-a3ee-811d5577ec5d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt" Oct 30 19:20:28.607464 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.607237 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls podName:d7dfc7a1-6b86-471f-b85b-d5ea756db9cb nodeName:}" failed. No retries permitted until 2026-10-30 19:20:29.107221703 +0000 UTC m=+31.980408991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fgx9q" (UID: "d7dfc7a1-6b86-471f-b85b-d5ea756db9cb") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:28.607464 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8e59f23-4988-41f7-b462-912ac6c7c125-serving-cert\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.607464 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:28.607464 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-installation-pull-secrets\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.607866 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56b5\" (UniqueName: \"kubernetes.io/projected/9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1-kube-api-access-t56b5\") pod \"kas-connection-checker-685895c7c9-qgdzh\" (UID: \"9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1\") " pod="kube-system/kas-connection-checker-685895c7c9-qgdzh" Oct 30 19:20:28.607866 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-stats-auth\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.607866 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-kube-api-access-s5qrf\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.607866 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607587 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-trusted-ca\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.607866 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607599 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dqf\" (UniqueName: \"kubernetes.io/projected/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-kube-api-access-44dqf\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.607866 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607625 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-trusted-ca\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.608212 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.607937 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-trusted-ca\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.608212 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.608137 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:28.609607 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.609586 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-serving-cert\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.609607 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.609598 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-image-registry-private-configuration\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.609707 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.609663 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-installation-pull-secrets\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.615543 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.615526 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 19:20:28.615708 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.615691 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 19:20:28.615785 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.615732 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-ww87v\"" Oct 30 19:20:28.615785 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.615777 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zfxvd\"" Oct 30 19:20:28.615874 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.615803 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 19:20:28.615874 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.615817 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 19:20:28.615874 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.615830 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vp447\"" Oct 30 19:20:28.615975 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.615884 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-g6588\"" Oct 30 19:20:28.615975 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.615898 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zqwsx\"" Oct 30 19:20:28.620793 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.620778 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-77k4g"] Oct 30 19:20:28.620906 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.620894 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt" Oct 30 19:20:28.623410 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.623384 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqlw\" (UniqueName: \"kubernetes.io/projected/b07766d9-9884-4f58-9c94-e59540953a06-kube-api-access-pdqlw\") pod \"kas-connection-checker-685895c7c9-c6bdc\" (UID: \"b07766d9-9884-4f58-9c94-e59540953a06\") " pod="kube-system/kas-connection-checker-685895c7c9-c6bdc" Oct 30 19:20:28.623600 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.623576 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpf8n\" (UniqueName: \"kubernetes.io/projected/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-kube-api-access-vpf8n\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:28.624808 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.624784 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-bound-sa-token\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.625636 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.625615 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghlh\" (UniqueName: \"kubernetes.io/projected/e6811b53-7458-4b8c-b4f9-9f46fb029eea-kube-api-access-9ghlh\") pod \"kas-connection-checker-685895c7c9-gt86s\" (UID: \"e6811b53-7458-4b8c-b4f9-9f46fb029eea\") " pod="kube-system/kas-connection-checker-685895c7c9-gt86s" Oct 30 19:20:28.625909 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.625891 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56b5\" (UniqueName: \"kubernetes.io/projected/9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1-kube-api-access-t56b5\") pod \"kas-connection-checker-685895c7c9-qgdzh\" (UID: \"9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1\") " pod="kube-system/kas-connection-checker-685895c7c9-qgdzh" Oct 30 19:20:28.625992 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.625977 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-kube-api-access-s5qrf\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:28.626223 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.626203 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dqf\" (UniqueName: \"kubernetes.io/projected/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-kube-api-access-44dqf\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.646987 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.646971 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hlqc9"] Oct 30 19:20:28.647089 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.647074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.661002 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.660985 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qgdzh"] Oct 30 19:20:28.661079 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661023 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c6bdc"] Oct 30 19:20:28.661079 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661041 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gt86s"] Oct 30 19:20:28.661079 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661055 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q"] Oct 30 19:20:28.661079 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661069 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9g8dh"] Oct 30 19:20:28.661222 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661083 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf497f4fc-s2vpv"] Oct 30 19:20:28.661222 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661097 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42"] Oct 30 19:20:28.661222 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661132 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt"] Oct 30 19:20:28.661222 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661142 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:28.661415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661147 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d984dbd48-qljg6"] Oct 30 19:20:28.661415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661282 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8ns8s"] Oct 30 19:20:28.661415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661311 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt"] Oct 30 19:20:28.661415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661331 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx"] Oct 30 19:20:28.661415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661362 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hlqc9"] Oct 30 19:20:28.661415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661387 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-77k4g"] Oct 30 19:20:28.661415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661414 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-58fgj"] Oct 30 19:20:28.661633 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661433 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk"] Oct 30 19:20:28.661633 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.661459 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56bc986487-ljfn2"] Oct 30 19:20:28.682840 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.682819 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.682840 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.682825 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.682971 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.682959 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 19:20:28.682971 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.682965 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.683046 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.682974 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.683046 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.683004 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:20:28.683046 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.683030 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:20:28.683166 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.683053 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:20:28.695163 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.695150 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qgdzh" Oct 30 19:20:28.708893 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.708877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-stats-auth\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.708938 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.708903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-trusted-ca\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.708938 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.708919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.709014 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.708937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1697d46a-5b9e-429a-986f-76e472a662a5-ca-trust-extracted\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.709014 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.708961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-registry-certificates\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.709014 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.708984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70b0df95-c5a5-48b5-b670-8382137780ea-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:28.709165 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.709165 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmpf\" (UniqueName: \"kubernetes.io/projected/2305471f-d5ac-43c7-b204-1650ae447ff9-kube-api-access-pkmpf\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:28.709165 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-installation-pull-secrets\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.709310 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-tmp\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.709310 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e8615b0-abab-426b-a0ee-893109807207-serving-cert\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.709310 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnlzn\" (UniqueName: \"kubernetes.io/projected/f8e59f23-4988-41f7-b462-912ac6c7c125-kube-api-access-fnlzn\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.709310 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-bound-sa-token\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.709488 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-service-ca-bundle\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.709488 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-snapshots\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.709488 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709389 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-image-registry-private-configuration\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.709488 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkf8\" (UniqueName: \"kubernetes.io/projected/605b308f-b6a9-427f-a365-c0f8b110da64-kube-api-access-2lkf8\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.709488 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709425 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1697d46a-5b9e-429a-986f-76e472a662a5-ca-trust-extracted\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.709488 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-default-certificate\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.709746 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm957\" (UniqueName: \"kubernetes.io/projected/2e8615b0-abab-426b-a0ee-893109807207-kube-api-access-wm957\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.709746 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8e59f23-4988-41f7-b462-912ac6c7c125-config\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.709746 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8xmf\" (UniqueName: \"kubernetes.io/projected/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-kube-api-access-n8xmf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.709746 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:28.709746 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:28.709746 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709621 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-tmp\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.710025 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709942 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-registry-certificates\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.710025 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.709955 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.710025 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710000 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-service-ca-bundle\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.710204 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710040 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-trusted-ca\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.710204 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710178 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-snapshots\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.710309 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.710363 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710305 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:28.710363 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.710465 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkqc\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-kube-api-access-lvkqc\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.710465 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710390 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2978\" (UniqueName: \"kubernetes.io/projected/6ac8874c-1de6-4a26-be0c-ad3a459c4d65-kube-api-access-g2978\") pod \"network-check-source-6f78d59f99-52vqt\" (UID: \"6ac8874c-1de6-4a26-be0c-ad3a459c4d65\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt" Oct 30 19:20:28.710563 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.710471 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:28.710563 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.710484 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf497f4fc-s2vpv: secret "image-registry-tls" not found Oct 30 19:20:28.710563 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.710525 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls podName:1697d46a-5b9e-429a-986f-76e472a662a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:29.210510595 +0000 UTC m=+32.083697877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls") pod "image-registry-5cf497f4fc-s2vpv" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5") : secret "image-registry-tls" not found Oct 30 19:20:28.710563 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710539 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8e59f23-4988-41f7-b462-912ac6c7c125-config\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.710778 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710628 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.710778 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.710778 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqt56\" (UniqueName: \"kubernetes.io/projected/c2f5e9c8-28aa-4956-a3ee-811d5577ec5d-kube-api-access-zqt56\") pod \"volume-data-source-validator-58c9f5f46d-fj5bt\" (UID: \"c2f5e9c8-28aa-4956-a3ee-811d5577ec5d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt" Oct 30 19:20:28.710778 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.710757 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:29.210740273 +0000 UTC m=+32.083927568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : configmap references non-existent config key: service-ca.crt Oct 30 19:20:28.710778 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.710760 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:20:28.711017 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.710786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8e59f23-4988-41f7-b462-912ac6c7c125-serving-cert\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.711017 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.710805 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:29.210792121 +0000 UTC m=+32.083979400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : secret "router-metrics-certs-default" not found Oct 30 19:20:28.712542 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.712517 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e8615b0-abab-426b-a0ee-893109807207-serving-cert\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.712918 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.712895 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-image-registry-private-configuration\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.713023 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.712982 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-default-certificate\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.713321 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.713286 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8e59f23-4988-41f7-b462-912ac6c7c125-serving-cert\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.713651 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.713630 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-stats-auth\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.713731 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.713677 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-installation-pull-secrets\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.715458 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.715436 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c6bdc" Oct 30 19:20:28.715870 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.715851 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hnk7h\"" Oct 30 19:20:28.715870 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.715858 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:20:28.716008 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.715902 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m2qt2\"" Oct 30 19:20:28.716008 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.715959 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:20:28.716008 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.715981 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lkdbb\"" Oct 30 19:20:28.727371 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.727244 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-bound-sa-token\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.728700 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.728672 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm957\" (UniqueName: \"kubernetes.io/projected/2e8615b0-abab-426b-a0ee-893109807207-kube-api-access-wm957\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.729179 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.729152 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqt56\" (UniqueName: \"kubernetes.io/projected/c2f5e9c8-28aa-4956-a3ee-811d5577ec5d-kube-api-access-zqt56\") pod \"volume-data-source-validator-58c9f5f46d-fj5bt\" (UID: \"c2f5e9c8-28aa-4956-a3ee-811d5577ec5d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt" Oct 30 19:20:28.729351 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.729334 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkf8\" (UniqueName: \"kubernetes.io/projected/605b308f-b6a9-427f-a365-c0f8b110da64-kube-api-access-2lkf8\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:28.729951 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.729932 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnlzn\" (UniqueName: \"kubernetes.io/projected/f8e59f23-4988-41f7-b462-912ac6c7c125-kube-api-access-fnlzn\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.730085 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.730070 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkqc\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-kube-api-access-lvkqc\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:28.744865 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.744833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gt86s" Oct 30 19:20:28.785152 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.784962 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:28.811777 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.811395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.811777 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.811421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:28.811777 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.811458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2978\" (UniqueName: \"kubernetes.io/projected/6ac8874c-1de6-4a26-be0c-ad3a459c4d65-kube-api-access-g2978\") pod \"network-check-source-6f78d59f99-52vqt\" (UID: \"6ac8874c-1de6-4a26-be0c-ad3a459c4d65\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt" Oct 30 19:20:28.811777 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.811504 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044847a9-7c73-421b-98ae-50bbb505b30a-tmp-dir\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.811777 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.811525 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dsf9\" (UniqueName: \"kubernetes.io/projected/79411cba-ba1d-41d6-b656-a661befaaa11-kube-api-access-9dsf9\") pod \"ingress-canary-hlqc9\" (UID: \"79411cba-ba1d-41d6-b656-a661befaaa11\") " pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:28.811777 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.811547 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.811777 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.811567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70b0df95-c5a5-48b5-b670-8382137780ea-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:28.811777 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.811584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.811777 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.811606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmpf\" (UniqueName: \"kubernetes.io/projected/2305471f-d5ac-43c7-b204-1650ae447ff9-kube-api-access-pkmpf\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:28.813633 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.813084 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70b0df95-c5a5-48b5-b670-8382137780ea-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:28.813633 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.813181 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.813633 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.813194 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79411cba-ba1d-41d6-b656-a661befaaa11-cert\") pod \"ingress-canary-hlqc9\" (UID: \"79411cba-ba1d-41d6-b656-a661befaaa11\") " pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:28.813633 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.813302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/044847a9-7c73-421b-98ae-50bbb505b30a-config-volume\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.813633 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.813331 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktbxp\" (UniqueName: \"kubernetes.io/projected/044847a9-7c73-421b-98ae-50bbb505b30a-kube-api-access-ktbxp\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.813633 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.813392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xmf\" (UniqueName: \"kubernetes.io/projected/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-kube-api-access-n8xmf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.813633 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.813421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:28.813633 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.813445 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:28.816397 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.814085 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:28.816397 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.814385 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:20:28.816397 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.814441 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert podName:70b0df95-c5a5-48b5-b670-8382137780ea nodeName:}" failed. No retries permitted until 2026-10-30 19:20:29.314423589 +0000 UTC m=+32.187610871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8ns8s" (UID: "70b0df95-c5a5-48b5-b670-8382137780ea") : secret "networking-console-plugin-cert" not found Oct 30 19:20:28.816397 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.814656 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:20:28.816397 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.814714 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls podName:2305471f-d5ac-43c7-b204-1650ae447ff9 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:29.314699861 +0000 UTC m=+32.187887142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dzhlx" (UID: "2305471f-d5ac-43c7-b204-1650ae447ff9") : secret "samples-operator-tls" not found Oct 30 19:20:28.818383 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.817894 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.831433 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.831376 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmpf\" (UniqueName: \"kubernetes.io/projected/2305471f-d5ac-43c7-b204-1650ae447ff9-kube-api-access-pkmpf\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:28.831771 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.831749 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2978\" (UniqueName: \"kubernetes.io/projected/6ac8874c-1de6-4a26-be0c-ad3a459c4d65-kube-api-access-g2978\") pod \"network-check-source-6f78d59f99-52vqt\" (UID: \"6ac8874c-1de6-4a26-be0c-ad3a459c4d65\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt" Oct 30 19:20:28.833870 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.833850 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xmf\" (UniqueName: \"kubernetes.io/projected/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-kube-api-access-n8xmf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.850729 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.850711 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt" Oct 30 19:20:28.870428 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.870407 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:28.890293 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.890157 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:28.914786 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.914759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79411cba-ba1d-41d6-b656-a661befaaa11-cert\") pod \"ingress-canary-hlqc9\" (UID: \"79411cba-ba1d-41d6-b656-a661befaaa11\") " pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:28.914914 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.914897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/044847a9-7c73-421b-98ae-50bbb505b30a-config-volume\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.914973 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.914929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktbxp\" (UniqueName: \"kubernetes.io/projected/044847a9-7c73-421b-98ae-50bbb505b30a-kube-api-access-ktbxp\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.915209 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.915187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044847a9-7c73-421b-98ae-50bbb505b30a-tmp-dir\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.915298 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.915233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dsf9\" (UniqueName: \"kubernetes.io/projected/79411cba-ba1d-41d6-b656-a661befaaa11-kube-api-access-9dsf9\") pod \"ingress-canary-hlqc9\" (UID: \"79411cba-ba1d-41d6-b656-a661befaaa11\") " pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:28.915352 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.915294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.917796 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.917764 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044847a9-7c73-421b-98ae-50bbb505b30a-tmp-dir\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.919163 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.919140 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:20:28.919253 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:28.919215 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls podName:044847a9-7c73-421b-98ae-50bbb505b30a nodeName:}" failed. No retries permitted until 2026-10-30 19:20:29.419195786 +0000 UTC m=+32.292383067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls") pod "dns-default-77k4g" (UID: "044847a9-7c73-421b-98ae-50bbb505b30a") : secret "dns-default-metrics-tls" not found Oct 30 19:20:28.919435 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.919416 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:28.921316 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.920964 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/044847a9-7c73-421b-98ae-50bbb505b30a-config-volume\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:28.924804 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.923866 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79411cba-ba1d-41d6-b656-a661befaaa11-cert\") pod \"ingress-canary-hlqc9\" (UID: \"79411cba-ba1d-41d6-b656-a661befaaa11\") " pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:28.937812 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.937783 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dsf9\" (UniqueName: \"kubernetes.io/projected/79411cba-ba1d-41d6-b656-a661befaaa11-kube-api-access-9dsf9\") pod \"ingress-canary-hlqc9\" (UID: \"79411cba-ba1d-41d6-b656-a661befaaa11\") " pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:28.941919 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:28.941898 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktbxp\" (UniqueName: \"kubernetes.io/projected/044847a9-7c73-421b-98ae-50bbb505b30a-kube-api-access-ktbxp\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:29.000210 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.000128 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:29.019603 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.019584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt" Oct 30 19:20:29.052262 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.052238 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c6bdc"] Oct 30 19:20:29.053505 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.053482 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qgdzh"] Oct 30 19:20:29.067057 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:20:29.066613 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07766d9_9884_4f58_9c94_e59540953a06.slice/crio-081fff3470d9d78fa24d1bfa5b348f66ea152f187defeb45210b52dc28e539e1 WatchSource:0}: Error finding container 081fff3470d9d78fa24d1bfa5b348f66ea152f187defeb45210b52dc28e539e1: Status 404 returned error can't find the container with id 081fff3470d9d78fa24d1bfa5b348f66ea152f187defeb45210b52dc28e539e1 Oct 30 19:20:29.090840 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.089007 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gt86s"] Oct 30 19:20:29.099060 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:20:29.099032 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6811b53_7458_4b8c_b4f9_9f46fb029eea.slice/crio-93ef467c2d3062bed30dffc9967529344be62f277d23114ef3258d6ed102b224 WatchSource:0}: Error finding container 93ef467c2d3062bed30dffc9967529344be62f277d23114ef3258d6ed102b224: Status 404 returned error can't find the container with id 93ef467c2d3062bed30dffc9967529344be62f277d23114ef3258d6ed102b224 Oct 30 19:20:29.116019 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.115793 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9g8dh"] Oct 30 19:20:29.118278 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.118047 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:29.118278 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.118145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:29.118556 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.118331 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:29.118556 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.118401 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls podName:d7dfc7a1-6b86-471f-b85b-d5ea756db9cb nodeName:}" failed. No retries permitted until 2026-10-30 19:20:30.118384636 +0000 UTC m=+32.991571916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fgx9q" (UID: "d7dfc7a1-6b86-471f-b85b-d5ea756db9cb") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:29.118556 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.118489 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:29.118556 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.118499 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d984dbd48-qljg6: secret "image-registry-tls" not found Oct 30 19:20:29.118556 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.118531 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls podName:a4cf2efa-495e-452f-b0a6-b17c1d54038b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:30.118520716 +0000 UTC m=+32.991707999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls") pod "image-registry-6d984dbd48-qljg6" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b") : secret "image-registry-tls" not found Oct 30 19:20:29.157321 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.157295 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt"] Oct 30 19:20:29.164270 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:20:29.164219 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f5e9c8_28aa_4956_a3ee_811d5577ec5d.slice/crio-2e5ce22040b681e6233ce5135eabc15cbd30f11374382760de79623d3a611f33 WatchSource:0}: Error finding container 2e5ce22040b681e6233ce5135eabc15cbd30f11374382760de79623d3a611f33: Status 404 returned error can't find the container with id 2e5ce22040b681e6233ce5135eabc15cbd30f11374382760de79623d3a611f33 Oct 30 19:20:29.193880 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.193760 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-58fgj"] Oct 30 19:20:29.203932 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:20:29.203751 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8615b0_abab_426b_a0ee_893109807207.slice/crio-319e69e1d1c44c2e7a889301a9d5fa0f0dbc30d7895ada90d3af66bd0803f1c3 WatchSource:0}: Error finding container 319e69e1d1c44c2e7a889301a9d5fa0f0dbc30d7895ada90d3af66bd0803f1c3: Status 404 returned error can't find the container with id 319e69e1d1c44c2e7a889301a9d5fa0f0dbc30d7895ada90d3af66bd0803f1c3 Oct 30 19:20:29.219392 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.218903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:29.219392 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.218948 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:29.219392 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.218971 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk"] Oct 30 19:20:29.219392 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.218974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:29.219392 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.219070 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:20:29.219392 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.219146 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:30.219127198 +0000 UTC m=+33.092314466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : secret "router-metrics-certs-default" not found Oct 30 19:20:29.219392 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.219219 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:29.219392 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.219231 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf497f4fc-s2vpv: secret "image-registry-tls" not found Oct 30 19:20:29.219392 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.219232 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:30.219220208 +0000 UTC m=+33.092407475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : configmap references non-existent config key: service-ca.crt Oct 30 19:20:29.219392 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.219275 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls podName:1697d46a-5b9e-429a-986f-76e472a662a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:30.219260202 +0000 UTC m=+33.092447490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls") pod "image-registry-5cf497f4fc-s2vpv" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5") : secret "image-registry-tls" not found Oct 30 19:20:29.223379 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:20:29.223349 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc69b2d3_eac5_43d1_86e4_36fa7cda7618.slice/crio-84d484bfe6adb6b57c4614debb9b8e18bacd0fa44229f59e00ed4b79faf88caa WatchSource:0}: Error finding container 84d484bfe6adb6b57c4614debb9b8e18bacd0fa44229f59e00ed4b79faf88caa: Status 404 returned error can't find the container with id 84d484bfe6adb6b57c4614debb9b8e18bacd0fa44229f59e00ed4b79faf88caa Oct 30 19:20:29.254610 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.254516 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42"] Oct 30 19:20:29.258790 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:20:29.258763 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e59f23_4988_41f7_b462_912ac6c7c125.slice/crio-6265281e2896a18db67c1982e6e5e24b57fd230fd6c956310ca73a8bf682b808 WatchSource:0}: Error finding container 6265281e2896a18db67c1982e6e5e24b57fd230fd6c956310ca73a8bf682b808: Status 404 returned error can't find the container with id 6265281e2896a18db67c1982e6e5e24b57fd230fd6c956310ca73a8bf682b808 Oct 30 19:20:29.319654 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.319626 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:29.319744 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.319675 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:29.319798 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.319783 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:20:29.319876 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.319864 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert podName:70b0df95-c5a5-48b5-b670-8382137780ea nodeName:}" failed. No retries permitted until 2026-10-30 19:20:30.319850115 +0000 UTC m=+33.193037396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8ns8s" (UID: "70b0df95-c5a5-48b5-b670-8382137780ea") : secret "networking-console-plugin-cert" not found Oct 30 19:20:29.319876 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.319865 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:20:29.319971 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.319929 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls podName:2305471f-d5ac-43c7-b204-1650ae447ff9 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:30.319912403 +0000 UTC m=+33.193099689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dzhlx" (UID: "2305471f-d5ac-43c7-b204-1650ae447ff9") : secret "samples-operator-tls" not found Oct 30 19:20:29.324021 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.323998 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hlqc9"] Oct 30 19:20:29.354500 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.354472 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt"] Oct 30 19:20:29.358086 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:20:29.358066 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac8874c_1de6_4a26_be0c_ad3a459c4d65.slice/crio-aaa4906984e878d19802e9cec1b144e4a2edb22510b6bdc772ea65548e8d0300 WatchSource:0}: Error finding container aaa4906984e878d19802e9cec1b144e4a2edb22510b6bdc772ea65548e8d0300: Status 404 returned error can't find the container with id aaa4906984e878d19802e9cec1b144e4a2edb22510b6bdc772ea65548e8d0300 Oct 30 19:20:29.420381 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.420359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:29.420509 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.420494 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:20:29.420550 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:29.420541 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls podName:044847a9-7c73-421b-98ae-50bbb505b30a nodeName:}" failed. No retries permitted until 2026-10-30 19:20:30.42052862 +0000 UTC m=+33.293715891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls") pod "dns-default-77k4g" (UID: "044847a9-7c73-421b-98ae-50bbb505b30a") : secret "dns-default-metrics-tls" not found Oct 30 19:20:29.536140 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.536094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:29.536209 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.536124 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:29.536511 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.536339 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:29.549891 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.549868 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sl59z\"" Oct 30 19:20:29.549971 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.549908 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:20:29.549971 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.549949 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-65nsn\"" Oct 30 19:20:29.550062 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.549999 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:20:29.550062 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.550013 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-77hs9\"" Oct 30 19:20:29.818930 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.818869 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt" event={"ID":"c2f5e9c8-28aa-4956-a3ee-811d5577ec5d","Type":"ContainerStarted","Data":"2e5ce22040b681e6233ce5135eabc15cbd30f11374382760de79623d3a611f33"} Oct 30 19:20:29.820441 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.820385 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" event={"ID":"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0","Type":"ContainerStarted","Data":"f42bef64e409d8b94cf6ac92c59c3501e86f1942c2ab126f1fcf3e5aceba16d3"} Oct 30 19:20:29.822517 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.821853 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gt86s" event={"ID":"e6811b53-7458-4b8c-b4f9-9f46fb029eea","Type":"ContainerStarted","Data":"2c3ed04ac82d31de41aae552f072d84dc96ff22e9ad61b395b8e49afd22684ca"} Oct 30 19:20:29.822517 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.821890 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gt86s" event={"ID":"e6811b53-7458-4b8c-b4f9-9f46fb029eea","Type":"ContainerStarted","Data":"93ef467c2d3062bed30dffc9967529344be62f277d23114ef3258d6ed102b224"} Oct 30 19:20:29.825076 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.825047 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qgdzh" event={"ID":"9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1","Type":"ContainerStarted","Data":"6b0fd7abd7b15117eff66c0b3314b68a0d7867007bd157b16b3c5ef78b17b053"} Oct 30 19:20:29.825213 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.825095 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qgdzh" event={"ID":"9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1","Type":"ContainerStarted","Data":"5225a087a7c998dc7ee863916ae21e5f4b75483581cf63b4d33904d12cd16332"} Oct 30 19:20:29.827088 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.827066 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hlqc9" event={"ID":"79411cba-ba1d-41d6-b656-a661befaaa11","Type":"ContainerStarted","Data":"0944cb6db616d7d6d34b10f0e0e51dea6e8c0e3e7f3b284d4c3f15754b480f14"} Oct 30 19:20:29.828333 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.828291 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" event={"ID":"dc69b2d3-eac5-43d1-86e4-36fa7cda7618","Type":"ContainerStarted","Data":"84d484bfe6adb6b57c4614debb9b8e18bacd0fa44229f59e00ed4b79faf88caa"} Oct 30 19:20:29.830072 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.830049 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c6bdc" event={"ID":"b07766d9-9884-4f58-9c94-e59540953a06","Type":"ContainerStarted","Data":"22ce0579b127db0a31bd5f5a54d99aef1f117199c6f368fba8ea9c150092642a"} Oct 30 19:20:29.830237 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.830087 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c6bdc" event={"ID":"b07766d9-9884-4f58-9c94-e59540953a06","Type":"ContainerStarted","Data":"081fff3470d9d78fa24d1bfa5b348f66ea152f187defeb45210b52dc28e539e1"} Oct 30 19:20:29.831638 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.831578 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt" event={"ID":"6ac8874c-1de6-4a26-be0c-ad3a459c4d65","Type":"ContainerStarted","Data":"aaa4906984e878d19802e9cec1b144e4a2edb22510b6bdc772ea65548e8d0300"} Oct 30 19:20:29.833161 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.833071 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" event={"ID":"f8e59f23-4988-41f7-b462-912ac6c7c125","Type":"ContainerStarted","Data":"6265281e2896a18db67c1982e6e5e24b57fd230fd6c956310ca73a8bf682b808"} Oct 30 19:20:29.835157 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.835118 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-58fgj" event={"ID":"2e8615b0-abab-426b-a0ee-893109807207","Type":"ContainerStarted","Data":"319e69e1d1c44c2e7a889301a9d5fa0f0dbc30d7895ada90d3af66bd0803f1c3"} Oct 30 19:20:29.862356 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.862307 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gt86s" podStartSLOduration=409.862293671 podStartE2EDuration="6m49.862293671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:29.860745041 +0000 UTC m=+32.733932331" watchObservedRunningTime="2026-10-30 19:20:29.862293671 +0000 UTC m=+32.735480962" Oct 30 19:20:29.900484 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:29.900336 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qgdzh" podStartSLOduration=409.900320771 podStartE2EDuration="6m49.900320771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:29.898397868 +0000 UTC m=+32.771585156" watchObservedRunningTime="2026-10-30 19:20:29.900320771 +0000 UTC m=+32.773508058" Oct 30 19:20:30.125088 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:30.124995 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:30.125264 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:30.125211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:30.125367 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.125347 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:30.125441 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.125369 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d984dbd48-qljg6: secret "image-registry-tls" not found Oct 30 19:20:30.125441 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.125426 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls podName:a4cf2efa-495e-452f-b0a6-b17c1d54038b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:32.125407121 +0000 UTC m=+34.998594392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls") pod "image-registry-6d984dbd48-qljg6" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b") : secret "image-registry-tls" not found Oct 30 19:20:30.125819 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.125804 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:30.125869 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.125854 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls podName:d7dfc7a1-6b86-471f-b85b-d5ea756db9cb nodeName:}" failed. No retries permitted until 2026-10-30 19:20:32.12584115 +0000 UTC m=+34.999028421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fgx9q" (UID: "d7dfc7a1-6b86-471f-b85b-d5ea756db9cb") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:30.226093 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:30.226064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:30.226265 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:30.226123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:30.226265 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.226222 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:30.226265 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.226244 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf497f4fc-s2vpv: secret "image-registry-tls" not found Oct 30 19:20:30.226415 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.226227 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:32.22621101 +0000 UTC m=+35.099398291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : configmap references non-existent config key: service-ca.crt Oct 30 19:20:30.226415 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.226302 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls podName:1697d46a-5b9e-429a-986f-76e472a662a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:32.226288234 +0000 UTC m=+35.099475502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls") pod "image-registry-5cf497f4fc-s2vpv" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5") : secret "image-registry-tls" not found Oct 30 19:20:30.226415 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:30.226319 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:30.226559 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.226428 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:20:30.226559 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.226450 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:32.22644353 +0000 UTC m=+35.099630798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : secret "router-metrics-certs-default" not found Oct 30 19:20:30.327669 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:30.327633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:30.327819 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:30.327696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:30.327919 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.327898 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:20:30.327980 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.327965 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls podName:2305471f-d5ac-43c7-b204-1650ae447ff9 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:32.327947571 +0000 UTC m=+35.201134859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dzhlx" (UID: "2305471f-d5ac-43c7-b204-1650ae447ff9") : secret "samples-operator-tls" not found Oct 30 19:20:30.328388 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.328365 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:20:30.328500 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.328432 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert podName:70b0df95-c5a5-48b5-b670-8382137780ea nodeName:}" failed. No retries permitted until 2026-10-30 19:20:32.328416972 +0000 UTC m=+35.201604241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8ns8s" (UID: "70b0df95-c5a5-48b5-b670-8382137780ea") : secret "networking-console-plugin-cert" not found Oct 30 19:20:30.428894 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:30.428825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:30.429038 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.429017 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:20:30.429197 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:30.429076 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls podName:044847a9-7c73-421b-98ae-50bbb505b30a nodeName:}" failed. No retries permitted until 2026-10-30 19:20:32.429060248 +0000 UTC m=+35.302247522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls") pod "dns-default-77k4g" (UID: "044847a9-7c73-421b-98ae-50bbb505b30a") : secret "dns-default-metrics-tls" not found Oct 30 19:20:31.337590 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.337553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:31.338888 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.337703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:31.338888 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:31.337899 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:20:31.338888 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:31.337960 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs podName:285534af-06ec-4171-93d8-2e3b57e1f85b nodeName:}" failed. No retries permitted until 2026-10-30 19:21:03.337942314 +0000 UTC m=+66.211129588 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs") pod "network-metrics-daemon-q7pb2" (UID: "285534af-06ec-4171-93d8-2e3b57e1f85b") : secret "metrics-daemon-secret" not found Oct 30 19:20:31.346926 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.346871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:31.383576 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.383448 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:31.439280 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.439252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:31.446429 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.446403 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:31.667975 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.667902 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:31.735175 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.735091 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-c6bdc" podStartSLOduration=411.735075256 podStartE2EDuration="6m51.735075256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:29.93821263 +0000 UTC m=+32.811399922" watchObservedRunningTime="2026-10-30 19:20:31.735075256 +0000 UTC m=+34.608262546" Oct 30 19:20:31.735501 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.735463 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vfmk2"] Oct 30 19:20:31.854015 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.853967 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt" event={"ID":"c2f5e9c8-28aa-4956-a3ee-811d5577ec5d","Type":"ContainerStarted","Data":"97c653957f1c48d23ebfc23ceefc73d8f55d9763a23a28ed295cd535444ff978"} Oct 30 19:20:31.856844 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.856777 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vfmk2" event={"ID":"6829a2b0-e64b-4b3c-9007-966545b88c8d","Type":"ContainerStarted","Data":"b7db49a53ebe6a45b52d4e0bc504b50b6ee6bb1df2cbcf99fae3e4b4d5c73cc8"} Oct 30 19:20:31.890459 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.890402 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt" podStartSLOduration=383.673933621 podStartE2EDuration="6m25.890384309s" totalImagesPullingTime="2.216450688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:31.888842991 +0000 UTC m=+34.762030281" watchObservedRunningTime="2026-10-30 19:20:31.890384309 +0000 UTC m=+34.763571600" Oct 30 19:20:31.998830 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:31.998760 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zrrw2"] Oct 30 19:20:32.003253 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:20:32.003217 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625ee392_6184_45a1_83a4_f577ee2062a0.slice/crio-d51f5d1f8e84527e21efd9c9d899412d11596d6abecc8aceb48d685a1e39d303 WatchSource:0}: Error finding container d51f5d1f8e84527e21efd9c9d899412d11596d6abecc8aceb48d685a1e39d303: Status 404 returned error can't find the container with id d51f5d1f8e84527e21efd9c9d899412d11596d6abecc8aceb48d685a1e39d303 Oct 30 19:20:32.168133 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:32.168080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:32.168305 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:32.168209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:32.168305 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.168228 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:32.168305 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.168249 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d984dbd48-qljg6: secret "image-registry-tls" not found Oct 30 19:20:32.168461 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.168320 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls podName:a4cf2efa-495e-452f-b0a6-b17c1d54038b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.168301964 +0000 UTC m=+39.041489245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls") pod "image-registry-6d984dbd48-qljg6" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b") : secret "image-registry-tls" not found Oct 30 19:20:32.168461 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.168388 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:32.168564 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.168463 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls podName:d7dfc7a1-6b86-471f-b85b-d5ea756db9cb nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.168447384 +0000 UTC m=+39.041634670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fgx9q" (UID: "d7dfc7a1-6b86-471f-b85b-d5ea756db9cb") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:32.270501 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:32.269557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:32.270501 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:32.269605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:32.270501 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:32.269634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:32.270501 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.269807 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:20:32.270501 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.269867 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.269849881 +0000 UTC m=+39.143037165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : secret "router-metrics-certs-default" not found Oct 30 19:20:32.270501 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.270287 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:32.270501 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.270304 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf497f4fc-s2vpv: secret "image-registry-tls" not found Oct 30 19:20:32.270501 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.270349 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls podName:1697d46a-5b9e-429a-986f-76e472a662a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.270334388 +0000 UTC m=+39.143521672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls") pod "image-registry-5cf497f4fc-s2vpv" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5") : secret "image-registry-tls" not found Oct 30 19:20:32.270501 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.270425 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.270414155 +0000 UTC m=+39.143601439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : configmap references non-existent config key: service-ca.crt Oct 30 19:20:32.370417 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:32.370382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:32.370853 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:32.370439 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:32.370853 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.370649 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:20:32.370853 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.370711 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls podName:2305471f-d5ac-43c7-b204-1650ae447ff9 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.370693683 +0000 UTC m=+39.243880970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dzhlx" (UID: "2305471f-d5ac-43c7-b204-1650ae447ff9") : secret "samples-operator-tls" not found Oct 30 19:20:32.371132 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.371094 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:20:32.371239 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.371173 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert podName:70b0df95-c5a5-48b5-b670-8382137780ea nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.371157743 +0000 UTC m=+39.244345020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8ns8s" (UID: "70b0df95-c5a5-48b5-b670-8382137780ea") : secret "networking-console-plugin-cert" not found Oct 30 19:20:32.471779 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:32.471745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:32.471928 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.471919 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:20:32.471989 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:32.471978 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls podName:044847a9-7c73-421b-98ae-50bbb505b30a nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.47196007 +0000 UTC m=+39.345147353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls") pod "dns-default-77k4g" (UID: "044847a9-7c73-421b-98ae-50bbb505b30a") : secret "dns-default-metrics-tls" not found Oct 30 19:20:32.864998 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:32.864920 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zrrw2" event={"ID":"625ee392-6184-45a1-83a4-f577ee2062a0","Type":"ContainerStarted","Data":"d51f5d1f8e84527e21efd9c9d899412d11596d6abecc8aceb48d685a1e39d303"} Oct 30 19:20:33.872385 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:33.872355 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/0.log" Oct 30 19:20:33.872816 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:33.872397 2573 generic.go:356] "Generic (PLEG): container finished" podID="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" containerID="f0312f4c5b46ab00cedc530e6f619a7ceaacd8e5db4e5ffe9ddc7f2c2b4f8650" exitCode=255 Oct 30 19:20:33.872816 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:33.872484 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" event={"ID":"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0","Type":"ContainerDied","Data":"f0312f4c5b46ab00cedc530e6f619a7ceaacd8e5db4e5ffe9ddc7f2c2b4f8650"} Oct 30 19:20:33.872816 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:33.872735 2573 scope.go:122] "RemoveContainer" containerID="f0312f4c5b46ab00cedc530e6f619a7ceaacd8e5db4e5ffe9ddc7f2c2b4f8650" Oct 30 19:20:34.881720 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:34.881692 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/1.log" Oct 30 19:20:34.883782 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:34.883764 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/0.log" Oct 30 19:20:34.884199 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:34.884180 2573 scope.go:122] "RemoveContainer" containerID="b5d4347479a7c97c79847e088687098f7c161fa6231cab27c845f8a624842a4e" Oct 30 19:20:34.884643 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:34.884434 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9g8dh_openshift-console-operator(1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" podUID="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" Oct 30 19:20:34.884691 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:34.884651 2573 generic.go:356] "Generic (PLEG): container finished" podID="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" containerID="b5d4347479a7c97c79847e088687098f7c161fa6231cab27c845f8a624842a4e" exitCode=255 Oct 30 19:20:34.884737 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:34.884713 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" event={"ID":"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0","Type":"ContainerDied","Data":"b5d4347479a7c97c79847e088687098f7c161fa6231cab27c845f8a624842a4e"} Oct 30 19:20:34.884789 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:34.884775 2573 scope.go:122] "RemoveContainer" containerID="f0312f4c5b46ab00cedc530e6f619a7ceaacd8e5db4e5ffe9ddc7f2c2b4f8650" Oct 30 19:20:35.890627 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:35.890604 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/1.log" Oct 30 19:20:35.891036 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:35.890985 2573 scope.go:122] "RemoveContainer" containerID="b5d4347479a7c97c79847e088687098f7c161fa6231cab27c845f8a624842a4e" Oct 30 19:20:35.891198 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:35.891166 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9g8dh_openshift-console-operator(1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" podUID="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" Oct 30 19:20:35.893209 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:35.893184 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hlqc9" event={"ID":"79411cba-ba1d-41d6-b656-a661befaaa11","Type":"ContainerStarted","Data":"c9b3991754f86663d75906ed3d5c3320eefa2303355084b24c5b202e2bbc1ab7"} Oct 30 19:20:35.896132 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:35.896060 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" event={"ID":"dc69b2d3-eac5-43d1-86e4-36fa7cda7618","Type":"ContainerStarted","Data":"9d878bd241f9c9ea8b5fc82e3e8f12a8b3dd90fed2994013a35bb87cab4a7d4f"} Oct 30 19:20:35.897698 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:35.897670 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt" event={"ID":"6ac8874c-1de6-4a26-be0c-ad3a459c4d65","Type":"ContainerStarted","Data":"5b5409d0a2d9929e022cff5e39f50a305adcf1bb94744e056b563908e7bb29cc"} Oct 30 19:20:35.900909 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:35.900846 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" event={"ID":"f8e59f23-4988-41f7-b462-912ac6c7c125","Type":"ContainerStarted","Data":"89332f550d212284a9adc7949f42f5deb7ce0bbb5775f8e5db5b3c9534f4355a"} Oct 30 19:20:35.902657 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:35.902596 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-58fgj" event={"ID":"2e8615b0-abab-426b-a0ee-893109807207","Type":"ContainerStarted","Data":"88ef7abcedff83cde2ba656b6d6e3482c333e09db5b0c7a1b3aed57f0feda0fa"} Oct 30 19:20:35.983328 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:35.975379 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-58fgj" podStartSLOduration=384.349338135 podStartE2EDuration="6m29.975361596s" totalImagesPullingTime="5.626023461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:35.973902511 +0000 UTC m=+38.847089804" watchObservedRunningTime="2026-10-30 19:20:35.975361596 +0000 UTC m=+38.848548889" Oct 30 19:20:36.021160 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.018745 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" podStartSLOduration=384.410008249 podStartE2EDuration="6m30.018725511s" totalImagesPullingTime="5.608717262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:36.018091665 +0000 UTC m=+38.891278979" watchObservedRunningTime="2026-10-30 19:20:36.018725511 +0000 UTC m=+38.891912805" Oct 30 19:20:36.057749 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.057587 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" podStartSLOduration=384.489535504 podStartE2EDuration="6m30.057573809s" totalImagesPullingTime="5.568038305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:36.05593005 +0000 UTC m=+38.929117342" watchObservedRunningTime="2026-10-30 19:20:36.057573809 +0000 UTC m=+38.930761099" Oct 30 19:20:36.121296 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.121246 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt" podStartSLOduration=380.824236747 podStartE2EDuration="6m27.121226791s" totalImagesPullingTime="6.296990044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:36.10046533 +0000 UTC m=+38.973652621" watchObservedRunningTime="2026-10-30 19:20:36.121226791 +0000 UTC m=+38.994414082" Oct 30 19:20:36.121611 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.121596 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m"] Oct 30 19:20:36.136604 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.136580 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m" Oct 30 19:20:36.141112 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.141053 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hlqc9" podStartSLOduration=2.615599136 podStartE2EDuration="8.141036475s" totalImagesPullingTime="5.525437339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:36.14079251 +0000 UTC m=+39.013979801" watchObservedRunningTime="2026-10-30 19:20:36.141036475 +0000 UTC m=+39.014223766" Oct 30 19:20:36.146078 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.146056 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m"] Oct 30 19:20:36.154557 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.154534 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:36.154682 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.154635 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:20:36.154740 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.154705 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5ffd5\"" Oct 30 19:20:36.210289 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.210220 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:36.210428 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.210294 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl2fc\" (UniqueName: \"kubernetes.io/projected/64a1dc94-4952-4113-b191-ca277a6a520a-kube-api-access-xl2fc\") pod \"migrator-66df5d496b-s9d5m\" (UID: \"64a1dc94-4952-4113-b191-ca277a6a520a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m" Oct 30 19:20:36.210428 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.210325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:36.210428 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.210347 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:36.210428 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.210368 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d984dbd48-qljg6: secret "image-registry-tls" not found Oct 30 19:20:36.210428 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.210418 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:36.210649 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.210419 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls podName:a4cf2efa-495e-452f-b0a6-b17c1d54038b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:44.210402522 +0000 UTC m=+47.083589802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls") pod "image-registry-6d984dbd48-qljg6" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b") : secret "image-registry-tls" not found Oct 30 19:20:36.210649 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.210535 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls podName:d7dfc7a1-6b86-471f-b85b-d5ea756db9cb nodeName:}" failed. No retries permitted until 2026-10-30 19:20:44.210522011 +0000 UTC m=+47.083709279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fgx9q" (UID: "d7dfc7a1-6b86-471f-b85b-d5ea756db9cb") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:36.247404 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.247369 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l55ql"] Oct 30 19:20:36.278093 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.277935 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l55ql"] Oct 30 19:20:36.278905 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.278041 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.311062 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.311032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:36.311190 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.311075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4scpw\" (UniqueName: \"kubernetes.io/projected/1741e885-17b0-42a2-abb0-92ae73771c59-kube-api-access-4scpw\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.311232 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.311208 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:20:36.311232 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.311213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1741e885-17b0-42a2-abb0-92ae73771c59-data-volume\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.311302 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.311250 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.311302 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.311290 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:44.311272243 +0000 UTC m=+47.184459516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : secret "router-metrics-certs-default" not found Oct 30 19:20:36.311368 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.311356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2fc\" (UniqueName: \"kubernetes.io/projected/64a1dc94-4952-4113-b191-ca277a6a520a-kube-api-access-xl2fc\") pod \"migrator-66df5d496b-s9d5m\" (UID: \"64a1dc94-4952-4113-b191-ca277a6a520a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m" Oct 30 19:20:36.311399 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.311382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1741e885-17b0-42a2-abb0-92ae73771c59-crio-socket\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.311432 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.311411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.311485 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.311474 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:36.311517 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.311498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:36.311682 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.311612 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:44.31159852 +0000 UTC m=+47.184785806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : configmap references non-existent config key: service-ca.crt Oct 30 19:20:36.311682 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.311618 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:36.311682 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.311631 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf497f4fc-s2vpv: secret "image-registry-tls" not found Oct 30 19:20:36.311682 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.311667 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls podName:1697d46a-5b9e-429a-986f-76e472a662a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:44.311659018 +0000 UTC m=+47.184846285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls") pod "image-registry-5cf497f4fc-s2vpv" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5") : secret "image-registry-tls" not found Oct 30 19:20:36.342041 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.342016 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2fc\" (UniqueName: \"kubernetes.io/projected/64a1dc94-4952-4113-b191-ca277a6a520a-kube-api-access-xl2fc\") pod \"migrator-66df5d496b-s9d5m\" (UID: \"64a1dc94-4952-4113-b191-ca277a6a520a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m" Oct 30 19:20:36.354423 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.354384 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:20:36.354599 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.354578 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qmnn8\"" Oct 30 19:20:36.357050 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.357032 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:20:36.411884 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.411842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:36.412006 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.411896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1741e885-17b0-42a2-abb0-92ae73771c59-crio-socket\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.412006 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.411923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.412006 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.411952 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.412009 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4scpw\" (UniqueName: \"kubernetes.io/projected/1741e885-17b0-42a2-abb0-92ae73771c59-kube-api-access-4scpw\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.412097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1741e885-17b0-42a2-abb0-92ae73771c59-data-volume\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.412149 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.412236 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.412251 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.412295 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm podName:1741e885-17b0-42a2-abb0-92ae73771c59 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.912279576 +0000 UTC m=+39.785466850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l55ql" (UID: "1741e885-17b0-42a2-abb0-92ae73771c59") : configmap "kube-rbac-proxy" not found Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.412311 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert podName:70b0df95-c5a5-48b5-b670-8382137780ea nodeName:}" failed. No retries permitted until 2026-10-30 19:20:44.4123029 +0000 UTC m=+47.285490169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8ns8s" (UID: "70b0df95-c5a5-48b5-b670-8382137780ea") : secret "networking-console-plugin-cert" not found Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.412332 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1741e885-17b0-42a2-abb0-92ae73771c59-crio-socket\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.412347 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.412385 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls podName:2305471f-d5ac-43c7-b204-1650ae447ff9 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:44.412372888 +0000 UTC m=+47.285560156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dzhlx" (UID: "2305471f-d5ac-43c7-b204-1650ae447ff9") : secret "samples-operator-tls" not found Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.412434 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.412467 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls podName:1741e885-17b0-42a2-abb0-92ae73771c59 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.912456434 +0000 UTC m=+39.785643707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l55ql" (UID: "1741e885-17b0-42a2-abb0-92ae73771c59") : secret "insights-runtime-extractor-tls" not found Oct 30 19:20:36.412651 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.412591 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1741e885-17b0-42a2-abb0-92ae73771c59-data-volume\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.430529 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.430509 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4scpw\" (UniqueName: \"kubernetes.io/projected/1741e885-17b0-42a2-abb0-92ae73771c59-kube-api-access-4scpw\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.470985 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.470930 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m" Oct 30 19:20:36.512849 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.512825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:36.512957 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.512917 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:20:36.513091 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.512968 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls podName:044847a9-7c73-421b-98ae-50bbb505b30a nodeName:}" failed. No retries permitted until 2026-10-30 19:20:44.51295117 +0000 UTC m=+47.386138467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls") pod "dns-default-77k4g" (UID: "044847a9-7c73-421b-98ae-50bbb505b30a") : secret "dns-default-metrics-tls" not found Oct 30 19:20:36.814173 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.813756 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m"] Oct 30 19:20:36.817654 ip-10-0-138-21 kubenswrapper[2573]: W1030 19:20:36.817625 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a1dc94_4952_4113_b191_ca277a6a520a.slice/crio-cc06f5e368974c52e61aa10e7245daadd3d5eb5f50f4589ad4f7aba8628da4d8 WatchSource:0}: Error finding container cc06f5e368974c52e61aa10e7245daadd3d5eb5f50f4589ad4f7aba8628da4d8: Status 404 returned error can't find the container with id cc06f5e368974c52e61aa10e7245daadd3d5eb5f50f4589ad4f7aba8628da4d8 Oct 30 19:20:36.907003 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.906969 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m" event={"ID":"64a1dc94-4952-4113-b191-ca277a6a520a","Type":"ContainerStarted","Data":"cc06f5e368974c52e61aa10e7245daadd3d5eb5f50f4589ad4f7aba8628da4d8"} Oct 30 19:20:36.908847 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.908809 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zrrw2" event={"ID":"625ee392-6184-45a1-83a4-f577ee2062a0","Type":"ContainerStarted","Data":"4f9a07bb2230ad39fbed1237d0db2b60e09090cc56419a1e4e01e7da04c7bcca"} Oct 30 19:20:36.915989 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.915967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.916118 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.916069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:36.916183 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.916147 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:20:36.916183 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.916169 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:20:36.916291 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.916213 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls podName:1741e885-17b0-42a2-abb0-92ae73771c59 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:37.916198524 +0000 UTC m=+40.789385809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l55ql" (UID: "1741e885-17b0-42a2-abb0-92ae73771c59") : secret "insights-runtime-extractor-tls" not found Oct 30 19:20:36.916546 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:36.916530 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm podName:1741e885-17b0-42a2-abb0-92ae73771c59 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:37.916519043 +0000 UTC m=+40.789706310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l55ql" (UID: "1741e885-17b0-42a2-abb0-92ae73771c59") : configmap "kube-rbac-proxy" not found Oct 30 19:20:36.951253 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:36.951208 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zrrw2" podStartSLOduration=35.079943216 podStartE2EDuration="38.95119438s" totalImagesPullingTime="3.871251164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:36.950917585 +0000 UTC m=+39.824104876" watchObservedRunningTime="2026-10-30 19:20:36.95119438 +0000 UTC m=+39.824381669" Oct 30 19:20:37.913002 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:37.912971 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:37.928177 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:37.928140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:37.928315 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:37.928250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:37.928409 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:37.928393 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:20:37.928466 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:37.928445 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls podName:1741e885-17b0-42a2-abb0-92ae73771c59 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:39.928433554 +0000 UTC m=+42.801620825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l55ql" (UID: "1741e885-17b0-42a2-abb0-92ae73771c59") : secret "insights-runtime-extractor-tls" not found Oct 30 19:20:37.928548 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:37.928531 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:20:37.928610 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:37.928588 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm podName:1741e885-17b0-42a2-abb0-92ae73771c59 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:39.928571498 +0000 UTC m=+42.801758780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l55ql" (UID: "1741e885-17b0-42a2-abb0-92ae73771c59") : configmap "kube-rbac-proxy" not found Oct 30 19:20:38.786207 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:38.786186 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:38.786320 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:38.786256 2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:38.786586 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:38.786575 2573 scope.go:122] "RemoveContainer" containerID="b5d4347479a7c97c79847e088687098f7c161fa6231cab27c845f8a624842a4e" Oct 30 19:20:38.786773 ip-10-0-138-21 kubenswrapper[2573]: E1030 19:20:38.786755 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9g8dh_openshift-console-operator(1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" podUID="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" Oct 30 19:20:38.919242 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:38.919208 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m" event={"ID":"64a1dc94-4952-4113-b191-ca277a6a520a","Type":"ContainerStarted","Data":"377e3ec2f037b2827e5f40b412f13099aa5cc00f8442f08f489f5e04ababcfe5"} Oct 30 19:20:38.920796 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:38.920766 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vfmk2" event={"ID":"6829a2b0-e64b-4b3c-9007-966545b88c8d","Type":"ContainerStarted","Data":"48915878cb31c5e7b137bfd68beff50b9c55c5d6e132ce13b4bdd6cca1ac4008"} Oct 30 19:20:38.957082 ip-10-0-138-21 kubenswrapper[2573]: I1030 19:20:38.956984 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m" podStartSLOduration=1.079765171 podStartE2EDuration="2.956970995s" totalImagesPullingTime="1.877205824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:38.955798385 +0000 UTC m=+41.828985673" watchObservedRunningTime="2026-10-30 19:20:38.956970995 +0000 UTC m=+41.830158286" Oct 30 19:20:39.001470 ip-10-0-138-21 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:20:39.009656 ip-10-0-138-21 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:20:39.009820 ip-10-0-138-21 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:20:39.009867 ip-10-0-138-21 systemd[1]: kubelet.service: Consumed 2.903s CPU time. Oct 30 19:20:39.018800 ip-10-0-138-21 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:20:39.134920 ip-10-0-138-21 kubenswrapper[4904]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:20:39.134920 ip-10-0-138-21 kubenswrapper[4904]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:20:39.134920 ip-10-0-138-21 kubenswrapper[4904]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:20:39.134920 ip-10-0-138-21 kubenswrapper[4904]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:20:39.137078 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.136991 4904 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:20:39.137078 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137080 4904 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137084 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137087 4904 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137089 4904 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137092 4904 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137095 4904 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137113 4904 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137118 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137122 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137125 4904 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137128 4904 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137131 4904 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137133 4904 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137136 4904 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137138 4904 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137142 4904 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137145 4904 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137147 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137150 4904 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137153 4904 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:20:39.137174 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137155 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137157 4904 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137160 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137164 4904 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137166 4904 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137169 4904 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137171 4904 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137174 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137177 4904 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137184 4904 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137187 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137190 4904 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137192 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137195 4904 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137198 4904 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137200 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137203 4904 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137206 4904 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137209 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:20:39.137711 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137211 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137214 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137217 4904 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137220 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137223 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137225 4904 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137228 4904 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137231 4904 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137234 4904 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137236 4904 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137239 4904 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137241 4904 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137244 4904 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137247 4904 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137250 4904 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137252 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137255 4904 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137258 4904 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137260 4904 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:20:39.138227 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137263 4904 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137266 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137269 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137271 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137276 4904 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137280 4904 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137283 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137286 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137289 4904 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137292 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137294 4904 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137296 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137299 4904 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137301 4904 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137304 4904 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137310 4904 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137313 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137315 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137318 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:20:39.138748 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137322 4904 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137327 4904 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137331 4904 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137334 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137337 4904 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137339 4904 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137342 4904 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137344 4904 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137347 4904 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137349 4904 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137352 4904 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137355 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137357 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137360 4904 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137362 4904 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137365 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137367 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137370 4904 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137372 4904 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137375 4904 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:20:39.139301 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137377 4904 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137380 4904 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137382 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137385 4904 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137388 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137391 4904 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137394 4904 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.137396 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138188 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138197 4904 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138200 4904 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138202 4904 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138205 4904 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138208 4904 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138211 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138214 4904 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138217 4904 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138220 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138222 4904 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:20:39.139858 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138225 4904 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138227 4904 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138230 4904 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138233 4904 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138236 4904 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138239 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138242 4904 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138244 4904 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138247 4904 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138251 4904 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138255 4904 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138258 4904 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138261 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138264 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138267 4904 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138269 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138272 4904 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138275 4904 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138277 4904 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:20:39.140460 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138280 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138282 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138286 4904 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138289 4904 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138291 4904 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138294 4904 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138296 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138299 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138301 4904 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138304 4904 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138307 4904 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138309 4904 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138312 4904 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138314 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138317 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138319 4904 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138321 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138325 4904 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138328 4904 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:20:39.141003 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138330 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138333 4904 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138335 4904 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138338 4904 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138341 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138343 4904 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138346 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138349 4904 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138352 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138354 4904 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138357 4904 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138360 4904 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138364 4904 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138366 4904 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138369 4904 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138371 4904 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138374 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138376 4904 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138379 4904 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138381 4904 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:20:39.141536 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138384 4904 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138386 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138388 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138391 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138394 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138396 4904 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138399 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138401 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138404 4904 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138406 4904 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138409 4904 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138411 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138414 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138417 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138419 4904 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138422 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138426 4904 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138430 4904 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138434 4904 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138439 4904 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:20:39.142071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138442 4904 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138446 4904 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138451 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138455 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138459 4904 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138462 4904 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138464 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138467 4904 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138469 4904 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138472 4904 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138474 4904 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138477 4904 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138481 4904 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138483 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138486 4904 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138507 4904 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138511 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138993 4904 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.138998 4904 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139001 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:20:39.142662 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139004 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139007 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139009 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139012 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139014 4904 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139017 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139019 4904 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139022 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139024 4904 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139027 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139030 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139032 4904 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139035 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139037 4904 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139040 4904 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139042 4904 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139045 4904 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139048 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139051 4904 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139053 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:20:39.143384 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139055 4904 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139058 4904 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139060 4904 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139063 4904 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139065 4904 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139067 4904 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139073 4904 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139075 4904 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139078 4904 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139081 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139084 4904 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139086 4904 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139089 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139092 4904 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139094 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139096 4904 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139114 4904 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139116 4904 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139120 4904 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139123 4904 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:20:39.144314 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139126 4904 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139129 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139131 4904 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139134 4904 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139137 4904 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139139 4904 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139141 4904 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139144 4904 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139147 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139149 4904 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139152 4904 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139154 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139157 4904 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139159 4904 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139162 4904 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139164 4904 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139167 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139169 4904 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139173 4904 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139175 4904 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:20:39.145260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139178 4904 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139180 4904 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139184 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139187 4904 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139189 4904 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139192 4904 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139194 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139197 4904 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139199 4904 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139204 4904 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139208 4904 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139211 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139214 4904 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139217 4904 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139219 4904 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139222 4904 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139225 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139227 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139230 4904 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:20:39.146187 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139232 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139235 4904 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139237 4904 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139240 4904 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139244 4904 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139247 4904 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139250 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139252 4904 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139255 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139257 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139260 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139263 4904 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139266 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139268 4904 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139272 4904 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139274 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139277 4904 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139280 4904 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139282 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:20:39.147071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139285 4904 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139287 4904 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139289 4904 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139292 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.139295 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139387 4904 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139395 4904 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139401 4904 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139406 4904 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139413 4904 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139418 4904 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139428 4904 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139434 4904 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139438 4904 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139441 4904 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139444 4904 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139447 4904 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139450 4904 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139453 4904 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139456 4904 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139459 4904 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139462 4904 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139465 4904 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139468 4904 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:20:39.148008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139473 4904 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139476 4904 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139479 4904 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139482 4904 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139486 4904 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139489 4904 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139493 4904 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139496 4904 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139499 4904 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139503 4904 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139505 4904 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139508 4904 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139514 4904 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139519 4904 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139522 4904 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139525 4904 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139528 4904 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139531 4904 flags.go:64] FLAG: --enable-server="true" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139534 4904 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139539 4904 flags.go:64] FLAG: --event-burst="100" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139542 4904 flags.go:64] FLAG: --event-qps="50" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139545 4904 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139548 4904 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139551 4904 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:20:39.149128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139555 4904 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139558 4904 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139561 4904 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139563 4904 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139566 4904 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139569 4904 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139572 4904 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139575 4904 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139579 4904 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139582 4904 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139585 4904 flags.go:64] FLAG: --feature-gates="" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139589 4904 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139592 4904 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139595 4904 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139598 4904 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139601 4904 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139604 4904 flags.go:64] FLAG: --help="false" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139607 4904 flags.go:64] FLAG: --hostname-override="ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139610 4904 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139613 4904 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139618 4904 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139621 4904 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139624 4904 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139627 4904 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:20:39.150351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139630 4904 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139632 4904 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139635 4904 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139638 4904 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139641 4904 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139644 4904 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139647 4904 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139649 4904 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139652 4904 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139655 4904 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139658 4904 flags.go:64] FLAG: --lock-file="" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139661 4904 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139663 4904 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139666 4904 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139672 4904 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139674 4904 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139678 4904 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139681 4904 flags.go:64] FLAG: --logging-format="text" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139684 4904 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139687 4904 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139690 4904 flags.go:64] FLAG: --manifest-url="" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139693 4904 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139697 4904 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139700 4904 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139714 4904 flags.go:64] FLAG: --max-pods="110" Oct 30 19:20:39.151445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139717 4904 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139720 4904 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139723 4904 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139727 4904 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139730 4904 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139733 4904 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139735 4904 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139746 4904 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139749 4904 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139752 4904 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139755 4904 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139758 4904 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139760 4904 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139763 4904 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139766 4904 flags.go:64] FLAG: --port="10250" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139770 4904 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139773 4904 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02294910dd9115a67" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139776 4904 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139779 4904 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139782 4904 flags.go:64] FLAG: --register-node="true" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139785 4904 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139789 4904 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139792 4904 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139796 4904 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139799 4904 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139802 4904 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:20:39.152261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139805 4904 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139808 4904 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139811 4904 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139813 4904 flags.go:64] FLAG: --runonce="false" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139816 4904 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139820 4904 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139823 4904 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139826 4904 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139829 4904 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139833 4904 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139836 4904 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139839 4904 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139842 4904 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139844 4904 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139847 4904 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139850 4904 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139853 4904 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139856 4904 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139859 4904 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139864 4904 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139867 4904 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139870 4904 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139873 4904 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139876 4904 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139879 4904 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:20:39.152956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139882 4904 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139885 4904 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139888 4904 flags.go:64] FLAG: --v="2" Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139892 4904 flags.go:64] FLAG: --version="false" Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139897 4904 flags.go:64] FLAG: --vmodule="" Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139901 4904 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.139904 4904 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140030 4904 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140034 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140037 4904 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140040 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140042 4904 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140045 4904 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140047 4904 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140050 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140053 4904 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140057 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140060 4904 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140062 4904 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140065 4904 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:20:39.153622 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140067 4904 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140071 4904 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140074 4904 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140077 4904 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140080 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140083 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140086 4904 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140089 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140091 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140094 4904 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140097 4904 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140115 4904 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140118 4904 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140121 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140123 4904 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140126 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140130 4904 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140133 4904 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140135 4904 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:20:39.154166 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140138 4904 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140140 4904 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140143 4904 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140145 4904 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140148 4904 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140151 4904 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140153 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140155 4904 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140159 4904 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140162 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140165 4904 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140168 4904 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140170 4904 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140173 4904 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140175 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140178 4904 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140180 4904 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140183 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140185 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140187 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:20:39.154691 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140190 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140192 4904 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140195 4904 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140197 4904 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140200 4904 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140202 4904 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140205 4904 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140207 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140210 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140215 4904 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140218 4904 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140221 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140223 4904 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140226 4904 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140228 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140231 4904 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140233 4904 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140236 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140238 4904 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140241 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:20:39.155294 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140243 4904 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140247 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140250 4904 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140252 4904 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140255 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140258 4904 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140260 4904 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140263 4904 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140265 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140276 4904 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140279 4904 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140282 4904 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140285 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140287 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140290 4904 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140292 4904 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140295 4904 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140297 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140300 4904 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:20:39.155856 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140303 4904 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140305 4904 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140309 4904 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140312 4904 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140314 4904 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140317 4904 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140319 4904 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140322 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140324 4904 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140327 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140329 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140332 4904 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140334 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140337 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140340 4904 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.140367 4904 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:20:39.156382 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140401 4904 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140406 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140409 4904 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140412 4904 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140414 4904 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140417 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140419 4904 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140422 4904 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140425 4904 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140427 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140429 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140432 4904 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140435 4904 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140437 4904 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140440 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140443 4904 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140445 4904 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140448 4904 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140452 4904 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:20:39.156849 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140454 4904 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140457 4904 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140459 4904 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140462 4904 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140465 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140467 4904 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140470 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140472 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140475 4904 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140478 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140480 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140483 4904 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140486 4904 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140489 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140492 4904 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140495 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140497 4904 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140500 4904 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140502 4904 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:20:39.157604 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140505 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140507 4904 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140510 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140512 4904 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140515 4904 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140517 4904 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140519 4904 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140523 4904 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140527 4904 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140530 4904 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140532 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140535 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140538 4904 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140541 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140543 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140546 4904 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140549 4904 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140552 4904 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140554 4904 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:20:39.158218 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140557 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140559 4904 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140562 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140564 4904 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140567 4904 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140569 4904 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140573 4904 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140575 4904 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140578 4904 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140580 4904 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140583 4904 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140586 4904 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140588 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140590 4904 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140593 4904 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140595 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140598 4904 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140600 4904 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140603 4904 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140605 4904 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:20:39.158708 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140607 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140610 4904 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140612 4904 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140615 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140617 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140621 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140623 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140626 4904 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140629 4904 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140631 4904 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140634 4904 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140636 4904 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140639 4904 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140641 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140644 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140647 4904 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140651 4904 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140654 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140659 4904 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:20:39.159260 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140661 4904 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140665 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140668 4904 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140670 4904 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140673 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140675 4904 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140678 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140680 4904 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140683 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.140685 4904 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.146129 4904 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.146150 4904 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.146177 4904 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.146184 4904 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146275 4904 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146284 4904 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146290 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146295 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146301 4904 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:20:39.159747 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146305 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146309 4904 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146314 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146319 4904 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146323 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146327 4904 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146331 4904 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146336 4904 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146340 4904 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146344 4904 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146349 4904 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146353 4904 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146357 4904 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146361 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146366 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146370 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146375 4904 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146380 4904 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146384 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146388 4904 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:20:39.160266 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146393 4904 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146399 4904 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146405 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146410 4904 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146422 4904 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146426 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146431 4904 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146435 4904 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146439 4904 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146443 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146448 4904 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146452 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146456 4904 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146461 4904 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146465 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146469 4904 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146481 4904 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146485 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146489 4904 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:20:39.160800 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146493 4904 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146497 4904 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146502 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146506 4904 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146510 4904 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146514 4904 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146519 4904 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146523 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146527 4904 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146532 4904 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146537 4904 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146541 4904 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146557 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146561 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146565 4904 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146570 4904 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146574 4904 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146578 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146590 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:20:39.161315 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146594 4904 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146599 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146603 4904 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146608 4904 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146612 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146617 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146621 4904 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146625 4904 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146630 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146634 4904 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146638 4904 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146642 4904 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146647 4904 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146651 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146655 4904 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146659 4904 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146664 4904 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146668 4904 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146672 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146676 4904 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:20:39.161803 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146681 4904 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146685 4904 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146690 4904 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146694 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146699 4904 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146703 4904 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146707 4904 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146711 4904 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146718 4904 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146723 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146727 4904 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146731 4904 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146745 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146750 4904 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146754 4904 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146758 4904 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146763 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146767 4904 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146771 4904 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:20:39.162419 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146775 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146780 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146784 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.146789 4904 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.146830 4904 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147096 4904 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147125 4904 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147130 4904 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147135 4904 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147139 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147143 4904 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147148 4904 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147152 4904 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147156 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147160 4904 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147165 4904 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147169 4904 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:20:39.162904 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147173 4904 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147177 4904 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147181 4904 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147185 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147189 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147193 4904 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147197 4904 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147201 4904 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147206 4904 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147220 4904 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147225 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147229 4904 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147234 4904 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147238 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147242 4904 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147246 4904 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147250 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147255 4904 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147260 4904 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147264 4904 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:20:39.163346 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147268 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147273 4904 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147277 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147281 4904 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147286 4904 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147290 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147294 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147298 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147302 4904 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147308 4904 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147315 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147320 4904 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147324 4904 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147329 4904 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147333 4904 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147337 4904 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147341 4904 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147345 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147350 4904 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:20:39.163893 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147354 4904 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147358 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147362 4904 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147373 4904 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147378 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147382 4904 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147386 4904 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147390 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147395 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147399 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147404 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147408 4904 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147412 4904 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147417 4904 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147421 4904 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147425 4904 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147429 4904 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147433 4904 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147438 4904 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147442 4904 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:20:39.164410 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147446 4904 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147450 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147455 4904 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147459 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147463 4904 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147467 4904 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147471 4904 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147475 4904 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147482 4904 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147488 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147494 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147498 4904 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147502 4904 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147506 4904 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147510 4904 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147514 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147526 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147531 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:20:39.164936 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147535 4904 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147541 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147545 4904 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147549 4904 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147554 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147558 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147562 4904 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147566 4904 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147570 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147575 4904 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147579 4904 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147584 4904 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147588 4904 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147593 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147597 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147602 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:20:39.165484 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147606 4904 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.147644 4904 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147705 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147712 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147716 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147721 4904 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147726 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147730 4904 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147734 4904 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147739 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147743 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147747 4904 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147752 4904 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147756 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147760 4904 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:20:39.165919 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147772 4904 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147779 4904 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147784 4904 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147788 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147793 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147797 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147801 4904 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147805 4904 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147810 4904 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147814 4904 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147818 4904 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147821 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147825 4904 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147829 4904 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147834 4904 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147838 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147842 4904 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147846 4904 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147850 4904 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147854 4904 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:20:39.166319 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147858 4904 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147862 4904 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147866 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147870 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147874 4904 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147881 4904 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147885 4904 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147890 4904 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147894 4904 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147898 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147902 4904 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147907 4904 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147912 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147927 4904 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147933 4904 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147937 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147941 4904 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147946 4904 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147957 4904 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147961 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:20:39.166846 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147966 4904 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147970 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147975 4904 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147979 4904 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147983 4904 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147987 4904 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147991 4904 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.147996 4904 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148001 4904 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148006 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148010 4904 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148015 4904 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148019 4904 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148022 4904 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148026 4904 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148030 4904 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148034 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148038 4904 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148043 4904 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:20:39.167416 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148047 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148051 4904 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148055 4904 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148060 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148064 4904 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148068 4904 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148073 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148087 4904 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148092 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148096 4904 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148119 4904 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148123 4904 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148128 4904 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148132 4904 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148136 4904 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148140 4904 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148144 4904 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148147 4904 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148151 4904 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148155 4904 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:20:39.167994 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148159 4904 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148163 4904 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148168 4904 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148172 4904 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148176 4904 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148180 4904 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148184 4904 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148192 4904 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148198 4904 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148204 4904 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148209 4904 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148213 4904 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148218 4904 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:39.148222 4904 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.148319 4904 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.149987 4904 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.150054 4904 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.150356 4904 server.go:1014] "Starting client certificate rotation" Oct 30 19:20:39.168780 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.150482 4904 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:20:39.169539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.150861 4904 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:14:58 +0000 UTC" deadline="2028-07-01 16:00:05.381687727 +0000 UTC" Oct 30 19:20:39.169539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.151033 4904 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14636h39m26.230659914s" Oct 30 19:20:39.169539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.152416 4904 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:20:39.169539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.153584 4904 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:20:39.169539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.155080 4904 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:20:39.169539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.156797 4904 log.go:25] "Validated CRI v1 image API" Oct 30 19:20:39.169539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.157540 4904 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:20:39.169539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.159380 4904 fs.go:135] Filesystem UUIDs: map[476531c9-3f6d-4320-a189-8ddf2182ae01:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f34170cd-8205-483d-b939-e47cefec7c36:/dev/nvme0n1p3] Oct 30 19:20:39.170242 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.159394 4904 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/081fff3470d9d78fa24d1bfa5b348f66ea152f187defeb45210b52dc28e539e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/081fff3470d9d78fa24d1bfa5b348f66ea152f187defeb45210b52dc28e539e1/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0944cb6db616d7d6d34b10f0e0e51dea6e8c0e3e7f3b284d4c3f15754b480f14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0944cb6db616d7d6d34b10f0e0e51dea6e8c0e3e7f3b284d4c3f15754b480f14/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/137711c73ab79fd6681042fe37d0099fce4d63786c4e67a33942951554f56db0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/137711c73ab79fd6681042fe37d0099fce4d63786c4e67a33942951554f56db0/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e5ce22040b681e6233ce5135eabc15cbd30f11374382760de79623d3a611f33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e5ce22040b681e6233ce5135eabc15cbd30f11374382760de79623d3a611f33/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/319e69e1d1c44c2e7a889301a9d5fa0f0dbc30d7895ada90d3af66bd0803f1c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/319e69e1d1c44c2e7a889301a9d5fa0f0dbc30d7895ada90d3af66bd0803f1c3/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5225a087a7c998dc7ee863916ae21e5f4b75483581cf63b4d33904d12cd16332/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5225a087a7c998dc7ee863916ae21e5f4b75483581cf63b4d33904d12cd16332/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59ed34f923ef2cb34c2de5e8ab3102794367aa63950b3ecd1ae4d80937dfc0d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59ed34f923ef2cb34c2de5e8ab3102794367aa63950b3ecd1ae4d80937dfc0d1/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6265281e2896a18db67c1982e6e5e24b57fd230fd6c956310ca73a8bf682b808/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6265281e2896a18db67c1982e6e5e24b57fd230fd6c956310ca73a8bf682b808/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/650acbdd35e837e582c04501a2cbe8dac53e5203ca5d566f3503a94a2cf1bc46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/650acbdd35e837e582c04501a2cbe8dac53e5203ca5d566f3503a94a2cf1bc46/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7abc4a27d1aa763cb4fa08cd00bb93c2cabcc3feb09667ea7daa12e6d7ae23f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7abc4a27d1aa763cb4fa08cd00bb93c2cabcc3feb09667ea7daa12e6d7ae23f5/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81cb7bada7e07c00a9ef3c3d469461a4e77d2336142fd49476735e211ba35710/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81cb7bada7e07c00a9ef3c3d469461a4e77d2336142fd49476735e211ba35710/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84d484bfe6adb6b57c4614debb9b8e18bacd0fa44229f59e00ed4b79faf88caa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84d484bfe6adb6b57c4614debb9b8e18bacd0fa44229f59e00ed4b79faf88caa/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93ef467c2d3062bed30dffc9967529344be62f277d23114ef3258d6ed102b224/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93ef467c2d3062bed30dffc9967529344be62f277d23114ef3258d6ed102b224/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aaa4906984e878d19802e9cec1b144e4a2edb22510b6bdc772ea65548e8d0300/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aaa4906984e878d19802e9cec1b144e4a2edb22510b6bdc772ea65548e8d0300/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7db49a53ebe6a45b52d4e0bc504b50b6ee6bb1df2cbcf99fae3e4b4d5c73cc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7db49a53ebe6a45b52d4e0bc504b50b6ee6bb1df2cbcf99fae3e4b4d5c73cc8/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9e1fc9bec15bfe68425b7bc1e9fa179cc7981afdff85b5fd2993f463f03047a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9e1fc9bec15bfe68425b7bc1e9fa179cc7981afdff85b5fd2993f463f03047a/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc06f5e368974c52e61aa10e7245daadd3d5eb5f50f4589ad4f7aba8628da4d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc06f5e368974c52e61aa10e7245daadd3d5eb5f50f4589ad4f7aba8628da4d8/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d51f5d1f8e84527e21efd9c9d899412d11596d6abecc8aceb48d685a1e39d303/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d51f5d1f8e84527e21efd9c9d899412d11596d6abecc8aceb48d685a1e39d303/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dda3d5b8ad4cca6cc86831b842cd8e6b948a745edda19e90592cce73f0273d6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dda3d5b8ad4cca6cc86831b842cd8e6b948a745edda19e90592cce73f0273d6d/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f42bef64e409d8b94cf6ac92c59c3501e86f1942c2ab126f1fcf3e5aceba16d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f42bef64e409d8b94cf6ac92c59c3501e86f1942c2ab126f1fcf3e5aceba16d3/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7ccea346eba8ce24bc800c05a1d30d659886a5e029fcfb289944af885ae6a96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7ccea346eba8ce24bc800c05a1d30d659886a5e029fcfb289944af885ae6a96/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/feb2c31699f11464ebd9050a87da5f7bf3c8d92fb4b9e9c849c6984486bb1696/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/feb2c31699f11464ebd9050a87da5f7bf3c8d92fb4b9e9c849c6984486bb1696/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffa802b816fc4ebcd018f4b21d01c2acab4fa247c028c3bf4e8051dcc08a0426/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffa802b816fc4ebcd018f4b21d01c2acab4fa247c028c3bf4e8051dcc08a0426/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/044847a9-7c73-421b-98ae-50bbb505b30a/volumes/kubernetes.io~projected/kube-api-access-ktbxp:{mountpoint:/var/lib/kubelet/pods/044847a9-7c73-421b-98ae-50bbb505b30a/volumes/kubernetes.io~projected/kube-api-access-ktbxp major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0926fd0c-2705-4d4d-afbd-7007b665bb4e/volumes/kubernetes.io~projected/kube-api-access-tbt4q:{mountpoint:/var/lib/kubelet/pods/0926fd0c-2705-4d4d-afbd-7007b665bb4e/volumes/kubernetes.io~projected/kube-api-access-tbt4q major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/volumes/kubernetes.io~projected/kube-api-access-44dqf:{mountpoint:/var/lib/kubelet/pods/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/volumes/kubernetes.io~projected/kube-api-access-44dqf major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~projected/kube-api-access-lvkqc:{mountpoint:/var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~projected/kube-api-access-lvkqc major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1741e885-17b0-42a2-abb0-92ae73771c59/volumes/kubernetes.io~projected/kube-api-access-4scpw:{mountpoint:/var/lib/kubelet/pods/1741e885-17b0-42a2-abb0-92ae73771c59/volumes/kubernetes.io~projected/kube-api-access-4scpw major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d809989-435e-4d38-807e-4bdbe414cf32/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1d809989-435e-4d38-807e-4bdbe414cf32/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2305471f-d5ac-43c7-b204-1650ae447ff9/volumes/kubernetes.io~projected/kube-api-access-pkmpf:{mountpoint:/var/lib/kubelet/pods/2305471f-d5ac-43c7-b204-1650ae447ff9/volumes/kubernetes.io~projected/kube-api-access-pkmpf major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/285534af-06ec-4171-93d8-2e3b57e1f85b/volumes/kubernetes.io~projected/kube-api-access-xdmf4:{mountpoint:/var/lib/kubelet/pods/285534af-06ec-4171-93d8-2e3b57e1f85b/volumes/kubernetes.io~projected/kube-api-access-xdmf4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e8615b0-abab-426b-a0ee-893109807207/volumes/kubernetes.io~projected/kube-api-access-wm957:{mountpoint:/var/lib/kubelet/pods/2e8615b0-abab-426b-a0ee-893109807207/volumes/kubernetes.io~projected/kube-api-access-wm957 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e8615b0-abab-426b-a0ee-893109807207/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2e8615b0-abab-426b-a0ee-893109807207/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44b56555-103f-4cc0-8c1c-507c3cc48880/volumes/kubernetes.io~projected/kube-api-access-knvfn:{mountpoint:/var/lib/kubelet/pods/44b56555-103f-4cc0-8c1c-507c3cc48880/volumes/kubernetes.io~projected/kube-api-access-knvfn major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1/volumes/kubernetes.io~projected/kube-api-access-gr4jx:{mountpoint:/var/lib/kubelet/pods/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1/volumes/kubernetes.io~projected/kube-api-access-gr4jx major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/605b308f-b6a9-427f-a365-c0f8b110da64/volumes/kubernetes.io~projected/kube-api-access-2lkf8:{mountpoint:/var/lib/kubelet/pods/605b308f-b6a9-427f-a365-c0f8b110da64/volumes/kubernetes.io~projected/kube-api-access-2lkf8 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/605b308f-b6a9-427f-a365-c0f8b110da64/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/605b308f-b6a9-427f-a365-c0f8b110da64/volumes/kubernetes.io~secret/default-certificate major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/605b308f-b6a9-427f-a365-c0f8b110da64/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/605b308f-b6a9-427f-a365-c0f8b110da64/volumes/kubernetes.io~secret/stats-auth major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625ee392-6184-45a1-83a4-f577ee2062a0/volumes/kubernetes.io~projected/kube-api-access-mhc7r:{mountpoint:/var/lib/kubelet/pods/625ee392-6184-45a1-83a4-f577ee2062a0/volumes/kubernetes.io~projected/kube-api-access-mhc7r major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64a1dc94-4952-4113-b191-ca277a6a520a/volumes/kubernetes.io~projected/kube-api-access-xl2fc:{mountpoint:/var/lib/kubelet/pods/64a1dc94-4952-4113-b191-ca277a6a520a/volumes/kubernetes.io~projected/kube-api-access-xl2fc major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6829a2b0-e64b-4b3c-9007-966545b88c8d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6829a2b0-e64b-4b3c-9007-966545b88c8d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ac8874c-1de6-4a26-be0c-ad3a459c4d65/volumes/kubernetes.io~projected/kube-api-access-g2978:{mountpoint:/var/lib/kubelet/pods/6ac8874c-1de6-4a26-be0c-ad3a459c4d65/volumes/kubernetes.io~projected/kube-api-access-g2978 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79411cba-ba1d-41d6-b656-a661befaaa11/volumes/kubernetes.io~projected/kube-api-access-9dsf9:{mountpoint:/var/lib/kubelet/pods/79411cba-ba1d-41d6-b656-a661befaaa11/volumes/kubernetes.io~projected/kube-api-access-9dsf9 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79411cba-ba1d-41d6-b656-a661befaaa11/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/79411cba-ba1d-41d6-b656-a661befaaa11/volumes/kubernetes.io~secret/cert major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1/volumes/kubernetes.io~projected/kube-api-access-t56b5:{mountpoint:/var/lib/kubelet/pods/9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1/volumes/kubernetes.io~projected/kube-api-access-t56b5 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~projected/kube-api-access-s5qrf:{mountpoint:/var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~projected/kube-api-access-s5qrf major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a94f5874-53f0-4be0-bb77-ae6d535356ef/volumes/kubernetes.io~projected/kube-api-access-j57kl:{mountpoint:/var/lib/kubelet/pods/a94f5874-53f0-4be0-bb77-ae6d535356ef/volumes/kubernetes.io~projected/kube-api-access-j57kl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b07766d9-9884-4f58-9c94-e59540953a06/volumes/kubernetes.io~projected/kube-api-access-pdqlw:{mountpoint:/var/lib/kubelet/pods/b07766d9-9884-4f58-9c94-e59540953a06/volumes/kubernetes.io~projected/kube-api-access-pdqlw major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e/volumes/kubernetes.io~projected/kube-api-access-gc66h:{mountpoint:/var/lib/kubelet/pods/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e/volumes/kubernetes.io~projected/kube-api-access-gc66h major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2f5e9c8-28aa-4956-a3ee-811d5577ec5d/volumes/kubernetes.io~projected/kube-api-access-zqt56:{mountpoint:/var/lib/kubelet/pods/c2f5e9c8-28aa-4956-a3ee-811d5577ec5d/volumes/kubernetes.io~projected/kube-api-access-zqt56 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f/volumes/kubernetes.io~projected/kube-api-access-fhgjf:{mountpoint:/var/lib/kubelet/pods/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f/volumes/kubernetes.io~projected/kube-api-access-fhgjf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb/volumes/kubernetes.io~projected/kube-api-access-vpf8n:{mountpoint:/var/lib/kubelet/pods/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb/volumes/kubernetes.io~projected/kube-api-access-vpf8n major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc69b2d3-eac5-43d1-86e4-36fa7cda7618/volumes/kubernetes.io~projected/kube-api-access-n8xmf:{mountpoint:/var/lib/kubelet/pods/dc69b2d3-eac5-43d1-86e4-36fa7cda7618/volumes/kubernetes.io~projected/kube-api-access-n8xmf major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc69b2d3-eac5-43d1-86e4-36fa7cda7618/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/dc69b2d3-eac5-43d1-86e4-36fa7cda7618/volumes/kubernetes.io~secret/serving-cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6811b53-7458-4b8c-b4f9-9f46fb029eea/volumes/kubernetes.io~projected/kube-api-access-9ghlh:{mountpoint:/var/lib/kubelet/pods/e6811b53-7458-4b8c-b4f9-9f46fb029eea/volumes/kubernetes.io~projected/kube-api-access-9ghlh major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f83325ae-1680-4213-94ca-12001250ffa9/volumes/kubernetes.io~projected/kube-api-access-8md4c:{mountpoint:/var/lib/kubelet/pods/f83325ae-1680-4213-94ca-12001250ffa9/volumes/kubernetes.io~projected/kube-api-access-8md4c major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8e59f23-4988-41f7-b462-912ac6c7c125/volumes/kubernetes.io~projected/kube-api-access-fnlzn:{mountpoint:/var/lib/kubelet/pods/f8e59f23-4988-41f7-b462-912ac6c7c125/volumes/kubernetes.io~projected/kube-api-access-fnlzn major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8e59f23-4988-41f7-b462-912ac6c7c125/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f8e59f23-4988-41f7-b462-912ac6c7c125/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feab0cd3-8408-4594-b808-95f5653188c2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/feab0cd3-8408-4594-b808-95f5653188c2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feab0cd3-8408-4594-b808-95f5653188c2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/feab0cd3-8408-4594-b808-95f5653188c2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feab0cd3-8408-4594-b808-95f5653188c2/volumes/kubernetes.io~projected/kube-api-access-p6p8r:{mountpoint:/var/lib/kubelet/pods/feab0cd3-8408-4594-b808-95f5653188c2/volumes/kubernetes.io~projected/kube-api-access-p6p8r major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cfed27f262153581214a37c8be4efeee0419ac30291749b934fad082ea669e49/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8da15a05e852c976ef35ab4b53a50a96e0b47c421de2375e91f809a20a7ab2c7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a06ceb61ea7bc86044e9b7096752e180727fb53865805bc57dcaee4efba6ceea/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/456b6ceba16d28dd25d14df16168a2265c9603aec782752517fda3a6c9d0d574/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/16225eb52e364fffac8777eebd920778781dc6f4706369bbc034190a80dd6dca/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/71cd11967c346fce4e15142c690c1bec151851b0366b3be66bf36d2a0dd5bbf2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a2a84fac8be415164f7929320d06b3d779b5c8a26c8be4a254e708f8e7705a69/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c6b4ce9dfb75a75ceb67fcd587cec6c272870760c409dae78186fe9df341604b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7fbd4a9b9413ebe5455b5635f6bbc7be6070e1a136e22d6c1d07f50bc1e544cd/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/806c4673ddb45cdc1fc2e86ddabe550cb75057858502d4e885b2673c2bb47486/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a8fb5d1ec483553b06554828f1b920d3cb6911a077ce8e88ed07f4fbc0a213de/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/c20ba26b00d312ba2a684699026812c934bb353553712e08897836f08780876d/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5e7db0fa47c0f119cabdc6f9fe2df80635c33093b178af793555c8e7dcd9a58a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6f92c6d484a0e8f5300325b437928e89ed6faef8914964b32e3d1dffe875bd3b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/aec16483856a064a78d8fea90819d7f6d49c6aa5d0a6baa72a5505cd0b68dcb9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4e88a33e3732cf5eb513441fe4943eac6fab036e4726c9742fca8a43e8b34d41/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4dee38ab8e215553a64b54e030db622bb476fc9acac8366470049f8671b27569/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cb8fa57c3b7e08808d23dff0aec6aae8bff8f132e8a34834971650e4feff9c60/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1ce5b194b1fbe6b9a67139e00972efa2d325eef75a294dc27330c5f5cbb661a0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3ec6d4ca863662a567628ce78cabdf54d5a021f5ae9a10fea556db36862d2d85/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b23ba51be78cc1e354deeff2a9fc55868941fe40e89d9705e53a60eeb060e615/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/7dca336df67b5e70dd736389003c03dabdeb5c928c5f1fd7c4cebb5fc99a0ca1/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/6de25380506c5d2ebe0836082d260955f1b906ebdbf2b0951840ace6829fd84f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/678d7bc86b3414be9a1f7294a906aca1f016c9f3e56d48e0602f588d7d9c1f5d/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/cfd75fa8972f70ed09715e13f2c55c23cbc78084b0ed10fcaad62cef0bd310d0/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/5e30e5ff30ce6250e7194567a7419f1031604d55adfeb156d15169d9eff96e92/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/854726fb1be0d90d958398ad8226e1dec3d87b8406b710a8eb3f0f0bf8c210f2/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/81cb8111f68a377da4a9cc67afafa2c9e2c48591fcd696723418b1e1b681a80d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e83bdb37bfb376ae3b2bcdc7286b32e25daebb38aa6ddb85ced5ded651ab8cfb/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/b35fce14edb8778787aaa16294ebf490446043a2316daa604e0d497e2de7aca5/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/da9001ea78d9c8a3328348290afc070a3e7667211c12e4ef4265eb275e06f206/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/2600218ad2d36a9193a257a37cadbd27fab6d520f4b1568b7d5535f3e1935053/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/092769fb769d46e4984726229626ad26ae17689f24ae0194be067e4ed0c45be8/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/1255ebdf57b2f07a16c252c7f82c27249e5f59ff25a2dbc472bbf9c6e3bb60fc/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/b5a851f0171e20500e8508c114a5e294796a464c6ca2869e97575ed4a89863fe/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/89dd924cce4e2b6eb2ed610ddd3f9b4aa1d215c145fb95fe7395fb2d643f04ea/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/d8ee9d25075a33b38689273e3ecbc39e5c69a140e870263752de77cd9b68be65/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/0d473f5ca471d7966c58380ea5ea8b751e467940868493d2bea0be7cfb230a6c/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d8d889d3c3961b9638b370d4fa06f3de98e0b515a2cd774b1ebe4d74aace22cb/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/23760d77e910a976e08f1d73dfe37bf65e670acbe3ee20f29d0d60eea66389f5/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/71a03669062336b955a9de73064bb161dab1ce72ace9fd7d661204d7336e49c6/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a7c660d1a18f759e1a45aff74b3f277454e50499c6d097ffe9823414f2f0b91f/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/000dea55e8eb88ed9797ff339294dc94ea8036dff216bec5b5b2fe71d3a2ad91/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/395d4865c1f068a7802e326b035a85d385e5d76577ed2c807cc0ad255e593a1e/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/74a3b7a8ab8b56caa33223a9b3e1b23bd68dc937d557e9df072fb53082f4ab92/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/c62e035bdeb718a993a50f6659835aa01f89ceb5f6db89e946a5f1946f8250f9/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/9b97b255742eb455d217be950b4daa10a3fc12c938e604cc30f13fa963ad6266/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/35ce6d06e76cb11524b28e1fefd2371028af700f177b8a09d885ba81ed1f926c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/d4d5749dfdc7c0a24e3920d2356f0774dcc37b477b5ed4bc28e7838f6cc497a8/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/1a8833ec37211e7fabed4121b21be5ddc11b3e74b1f84979bbe8a5a7f04f8424/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/37371680d96e47cdbb1ca3c624463623af80e8cf8b54d45da80eb89e860735b7/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/1d28440b305370d815d5e7cbc127cc1316db7d67a618048d29eb95392b990f76/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/acc1768a02e24d4d3cb281ad271df15b4d2dd29b5c32cbf15e86186353c723c4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/21b627ab39091dde67981babfae1b0652edc955da10b9ec6c88972e8170a63a8/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3af24406b3fd002b6ac84cf869d3cc06affd29952377b297c4e07ae08b6a5693/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/21e8aa372f039c0e3d14d91f92fd182bb9218821329991dc0ef8b1f28f58c4ea/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.166798 4904 manager.go:217] Machine: {Timestamp:2026-10-30 19:20:39.166337552 +0000 UTC m=+0.079942161 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102204 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27928f3b602256eadb2787e35717d0 SystemUUID:ec27928f-3b60-2256-eadb-2787e35717d0 BootID:b3f484a7-75f2-4fea-87f6-83eac4cf4b49 Filesystems:[{Device:/var/lib/kubelet/pods/f83325ae-1680-4213-94ca-12001250ffa9/volumes/kubernetes.io~projected/kube-api-access-8md4c DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e8615b0-abab-426b-a0ee-893109807207/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2e8615b0-abab-426b-a0ee-893109807207/volumes/kubernetes.io~projected/kube-api-access-wm957 DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93ef467c2d3062bed30dffc9967529344be62f277d23114ef3258d6ed102b224/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/044847a9-7c73-421b-98ae-50bbb505b30a/volumes/kubernetes.io~projected/kube-api-access-ktbxp DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84d484bfe6adb6b57c4614debb9b8e18bacd0fa44229f59e00ed4b79faf88caa/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6829a2b0-e64b-4b3c-9007-966545b88c8d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64a1dc94-4952-4113-b191-ca277a6a520a/volumes/kubernetes.io~projected/kube-api-access-xl2fc DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/605b308f-b6a9-427f-a365-c0f8b110da64/volumes/kubernetes.io~projected/kube-api-access-2lkf8 DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e5ce22040b681e6233ce5135eabc15cbd30f11374382760de79623d3a611f33/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc06f5e368974c52e61aa10e7245daadd3d5eb5f50f4589ad4f7aba8628da4d8/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7abc4a27d1aa763cb4fa08cd00bb93c2cabcc3feb09667ea7daa12e6d7ae23f5/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2f5e9c8-28aa-4956-a3ee-811d5577ec5d/volumes/kubernetes.io~projected/kube-api-access-zqt56 DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6ac8874c-1de6-4a26-be0c-ad3a459c4d65/volumes/kubernetes.io~projected/kube-api-access-g2978 DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f42bef64e409d8b94cf6ac92c59c3501e86f1942c2ab126f1fcf3e5aceba16d3/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb/volumes/kubernetes.io~projected/kube-api-access-vpf8n DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81cb7bada7e07c00a9ef3c3d469461a4e77d2336142fd49476735e211ba35710/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/feab0cd3-8408-4594-b808-95f5653188c2/volumes/kubernetes.io~projected/kube-api-access-p6p8r DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8e59f23-4988-41f7-b462-912ac6c7c125/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/605b308f-b6a9-427f-a365-c0f8b110da64/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2305471f-d5ac-43c7-b204-1650ae447ff9/volumes/kubernetes.io~projected/kube-api-access-pkmpf DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0926fd0c-2705-4d4d-afbd-7007b665bb4e/volumes/kubernetes.io~projected/kube-api-access-tbt4q DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6265281e2896a18db67c1982e6e5e24b57fd230fd6c956310ca73a8bf682b808/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59ed34f923ef2cb34c2de5e8ab3102794367aa63950b3ecd1ae4d80937dfc0d1/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dc69b2d3-eac5-43d1-86e4-36fa7cda7618/volumes/kubernetes.io~projected/kube-api-access-n8xmf DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~projected/kube-api-access-s5qrf DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/volumes/kubernetes.io~projected/kube-api-access-44dqf DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aaa4906984e878d19802e9cec1b144e4a2edb22510b6bdc772ea65548e8d0300/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1741e885-17b0-42a2-abb0-92ae73771c59/volumes/kubernetes.io~projected/kube-api-access-4scpw DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/650acbdd35e837e582c04501a2cbe8dac53e5203ca5d566f3503a94a2cf1bc46/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b07766d9-9884-4f58-9c94-e59540953a06/volumes/kubernetes.io~projected/kube-api-access-pdqlw DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a94f5874-53f0-4be0-bb77-ae6d535356ef/volumes/kubernetes.io~projected/kube-api-access-j57kl DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5225a087a7c998dc7ee863916ae21e5f4b75483581cf63b4d33904d12cd16332/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f/volumes/kubernetes.io~projected/kube-api-access-fhgjf DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d51f5d1f8e84527e21efd9c9d899412d11596d6abecc8aceb48d685a1e39d303/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dda3d5b8ad4cca6cc86831b842cd8e6b948a745edda19e90592cce73f0273d6d/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6811b53-7458-4b8c-b4f9-9f46fb029eea/volumes/kubernetes.io~projected/kube-api-access-9ghlh DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1/volumes/kubernetes.io~projected/kube-api-access-t56b5 DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/625ee392-6184-45a1-83a4-f577ee2062a0/volumes/kubernetes.io~projected/kube-api-access-mhc7r DeviceMajor:0 DeviceMinor:292 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/137711c73ab79fd6681042fe37d0099fce4d63786c4e67a33942951554f56db0/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7ccea346eba8ce24bc800c05a1d30d659886a5e029fcfb289944af885ae6a96/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7db49a53ebe6a45b52d4e0bc504b50b6ee6bb1df2cbcf99fae3e4b4d5c73cc8/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes/kubernetes.io~projected/kube-api-access-lvkqc DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/319e69e1d1c44c2e7a889301a9d5fa0f0dbc30d7895ada90d3af66bd0803f1c3/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1/volumes/kubernetes.io~projected/kube-api-access-gr4jx DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffa802b816fc4ebcd018f4b21d01c2acab4fa247c028c3bf4e8051dcc08a0426/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc69b2d3-eac5-43d1-86e4-36fa7cda7618/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/285534af-06ec-4171-93d8-2e3b57e1f85b/volumes/kubernetes.io~projected/kube-api-access-xdmf4 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/605b308f-b6a9-427f-a365-c0f8b110da64/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79411cba-ba1d-41d6-b656-a661befaaa11/volumes/kubernetes.io~projected/kube-api-access-9dsf9 DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/feb2c31699f11464ebd9050a87da5f7bf3c8d92fb4b9e9c849c6984486bb1696/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/feab0cd3-8408-4594-b808-95f5653188c2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1d809989-435e-4d38-807e-4bdbe414cf32/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e/volumes/kubernetes.io~projected/kube-api-access-gc66h DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44b56555-103f-4cc0-8c1c-507c3cc48880/volumes/kubernetes.io~projected/kube-api-access-knvfn DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/081fff3470d9d78fa24d1bfa5b348f66ea152f187defeb45210b52dc28e539e1/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8e59f23-4988-41f7-b462-912ac6c7c125/volumes/kubernetes.io~projected/kube-api-access-fnlzn DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/feab0cd3-8408-4594-b808-95f5653188c2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79411cba-ba1d-41d6-b656-a661befaaa11/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:239 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9e1fc9bec15bfe68425b7bc1e9fa179cc7981afdff85b5fd2993f463f03047a/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0944cb6db616d7d6d34b10f0e0e51dea6e8c0e3e7f3b284d4c3f15754b480f14/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:081fff3470d9d78 MacAddress:02:3e:dd:b8:74:91 Speed:10000 Mtu:8901} {Name:0944cb6db616d7d MacAddress:4e:da:bd:1a:2a:4b Speed:10000 Mtu:8901} {Name:2e5ce22040b681e MacAddress:1e:30:a9:ce:ab:b3 Speed:10000 Mtu:8901} {Name:319e69e1d1c44c2 MacAddress:16:3b:a3:3a:ec:ac Speed:10000 Mtu:8901} {Name:5225a087a7c998d MacAddress:b2:3d:bf:e5:97:03 Speed:10000 Mtu:8901} {Name:6265281e2896a18 MacAddress:ba:3b:10:16:3d:50 Speed:10000 Mtu:8901} {Name:84d484bfe6adb6b MacAddress:1a:ec:52:f8:f8:1d Speed:10000 Mtu:8901} {Name:93ef467c2d3062b MacAddress:3a:46:2e:30:39:a6 Speed:10000 Mtu:8901} {Name:aaa4906984e878d MacAddress:b2:5f:8f:8a:1e:98 Speed:10000 Mtu:8901} {Name:b7db49a53ebe6a4 MacAddress:b2:a7:c9:3c:17:11 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ec:44:90:0b Speed:0 Mtu:9001} {Name:br-int MacAddress:be:bf:fa:6c:19:05 Speed:0 Mtu:8901} {Name:cc06f5e368974c5 MacAddress:26:db:70:ac:a4:a9 Speed:10000 Mtu:8901} {Name:d51f5d1f8e84527 MacAddress:8a:43:c1:9a:44:b7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ec:44:90:0b Speed:0 Mtu:9001} {Name:f42bef64e409d8b MacAddress:f6:65:8b:e8:3a:cf Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ea:71:7d:2e:d3:89 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:de:19:63:fc:c6:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.167157 4904 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.167198 4904 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.167531 4904 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.167549 4904 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-21.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.168342 4904 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.168355 4904 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.168374 4904 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.168402 4904 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.168431 4904 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.169215 4904 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.169630 4904 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.169679 4904 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.169695 4904 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.169705 4904 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.169714 4904 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.169739 4904 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.170448 4904 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:20:39.170600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.170458 4904 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:20:39.171755 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.170865 4904 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:20:39.172958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.172939 4904 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:20:39.172958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.172950 4904 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:20:39.173122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173060 4904 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:20:39.173254 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173241 4904 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:20:39.173322 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173270 4904 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:20:39.173387 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173375 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:20:39.173439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173394 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:20:39.173439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173404 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:20:39.173439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173411 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:20:39.173439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173419 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:20:39.173439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173424 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:20:39.173439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173433 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:20:39.173439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173440 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:20:39.173695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173447 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:20:39.173695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173453 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:20:39.173695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173469 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:20:39.173695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173479 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:20:39.173695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173497 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:20:39.173695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.173502 4904 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:20:39.176841 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.176828 4904 server.go:1263] "Started kubelet" Oct 30 19:20:39.176929 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.176890 4904 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:20:39.177350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.176926 4904 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:20:39.177350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.177271 4904 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:20:39.177394 ip-10-0-138-21 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:20:39.177641 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.177613 4904 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:20:39.178228 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.178210 4904 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:20:39.182662 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.182635 4904 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.182655 4904 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.182859 4904 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.182873 4904 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:39.182871 4904 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found" Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.183027 4904 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.183380 4904 factory.go:55] Registering systemd factory Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.183396 4904 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.183760 4904 factory.go:153] Registering CRI-O factory Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.183777 4904 factory.go:223] Registration of the crio container factory successfully Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.183824 4904 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.183841 4904 factory.go:103] Registering Raw factory Oct 30 19:20:39.184186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.183855 4904 manager.go:1199] Started watching for new ooms in manager Oct 30 19:20:39.184824 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.184255 4904 manager.go:319] Starting recovery of all containers Oct 30 19:20:39.186529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.182713 4904 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:15:00 +0000 UTC" deadline="2028-06-21 12:38:39.427624859 +0000 UTC" Oct 30 19:20:39.188170 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188148 4904 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14393h18m0.23948308s" Oct 30 19:20:39.188170 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.187847 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="285534af-06ec-4171-93d8-2e3b57e1f85b" volumeName="kubernetes.io/projected/285534af-06ec-4171-93d8-2e3b57e1f85b-kube-api-access-xdmf4" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188192 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="044847a9-7c73-421b-98ae-50bbb505b30a" volumeName="kubernetes.io/projected/044847a9-7c73-421b-98ae-50bbb505b30a-kube-api-access-ktbxp" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188212 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1697d46a-5b9e-429a-986f-76e472a662a5" volumeName="kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-installation-pull-secrets" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188226 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d809989-435e-4d38-807e-4bdbe414cf32" volumeName="kubernetes.io/configmap/1d809989-435e-4d38-807e-4bdbe414cf32-konnectivity-ca" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188238 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2305471f-d5ac-43c7-b204-1650ae447ff9" volumeName="kubernetes.io/projected/2305471f-d5ac-43c7-b204-1650ae447ff9-kube-api-access-pkmpf" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188249 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="605b308f-b6a9-427f-a365-c0f8b110da64" volumeName="kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-default-certificate" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188260 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8615b0-abab-426b-a0ee-893109807207" volumeName="kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-service-ca-bundle" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188280 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1" volumeName="kubernetes.io/projected/9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1-kube-api-access-t56b5" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188295 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8e59f23-4988-41f7-b462-912ac6c7c125" volumeName="kubernetes.io/projected/f8e59f23-4988-41f7-b462-912ac6c7c125-kube-api-access-fnlzn" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188308 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8615b0-abab-426b-a0ee-893109807207" volumeName="kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-tmp" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188320 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64a1dc94-4952-4113-b191-ca277a6a520a" volumeName="kubernetes.io/projected/64a1dc94-4952-4113-b191-ca277a6a520a-kube-api-access-xl2fc" seLinuxMountContext="" Oct 30 19:20:39.188331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188331 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6829a2b0-e64b-4b3c-9007-966545b88c8d" volumeName="kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188343 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e" volumeName="kubernetes.io/projected/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-kube-api-access-gc66h" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188357 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feab0cd3-8408-4594-b808-95f5653188c2" volumeName="kubernetes.io/projected/feab0cd3-8408-4594-b808-95f5653188c2-kube-api-access-p6p8r" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188374 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" volumeName="kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-trusted-ca" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188386 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1697d46a-5b9e-429a-986f-76e472a662a5" volumeName="kubernetes.io/empty-dir/1697d46a-5b9e-429a-986f-76e472a662a5-ca-trust-extracted" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188397 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc69b2d3-eac5-43d1-86e4-36fa7cda7618" volumeName="kubernetes.io/configmap/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-config" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188408 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44b56555-103f-4cc0-8c1c-507c3cc48880" volumeName="kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188434 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79411cba-ba1d-41d6-b656-a661befaaa11" volumeName="kubernetes.io/secret/79411cba-ba1d-41d6-b656-a661befaaa11-cert" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188448 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e" volumeName="kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-env-overrides" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188459 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d09042e-9b40-41d8-803d-8d4a0bbf0fd1" volumeName="kubernetes.io/empty-dir/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-tmp-dir" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188470 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a94f5874-53f0-4be0-bb77-ae6d535356ef" volumeName="kubernetes.io/projected/a94f5874-53f0-4be0-bb77-ae6d535356ef-kube-api-access-j57kl" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188483 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6811b53-7458-4b8c-b4f9-9f46fb029eea" volumeName="kubernetes.io/projected/e6811b53-7458-4b8c-b4f9-9f46fb029eea-kube-api-access-9ghlh" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188495 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="044847a9-7c73-421b-98ae-50bbb505b30a" volumeName="kubernetes.io/empty-dir/044847a9-7c73-421b-98ae-50bbb505b30a-tmp-dir" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188517 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8615b0-abab-426b-a0ee-893109807207" volumeName="kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188530 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625ee392-6184-45a1-83a4-f577ee2062a0" volumeName="kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188541 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4cf2efa-495e-452f-b0a6-b17c1d54038b" volumeName="kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-installation-pull-secrets" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188569 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" volumeName="kubernetes.io/projected/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-kube-api-access-44dqf" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188587 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1697d46a-5b9e-429a-986f-76e472a662a5" volumeName="kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-trusted-ca" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188609 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1741e885-17b0-42a2-abb0-92ae73771c59" volumeName="kubernetes.io/projected/1741e885-17b0-42a2-abb0-92ae73771c59-kube-api-access-4scpw" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188620 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44b56555-103f-4cc0-8c1c-507c3cc48880" volumeName="kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-binary-copy" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188632 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d09042e-9b40-41d8-803d-8d4a0bbf0fd1" volumeName="kubernetes.io/projected/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-kube-api-access-gr4jx" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188644 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4cf2efa-495e-452f-b0a6-b17c1d54038b" volumeName="kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-certificates" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188656 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" volumeName="kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-config" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188667 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1741e885-17b0-42a2-abb0-92ae73771c59" volumeName="kubernetes.io/empty-dir/1741e885-17b0-42a2-abb0-92ae73771c59-data-volume" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188679 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b07766d9-9884-4f58-9c94-e59540953a06" volumeName="kubernetes.io/projected/b07766d9-9884-4f58-9c94-e59540953a06-kube-api-access-pdqlw" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188691 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1697d46a-5b9e-429a-986f-76e472a662a5" volumeName="kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-bound-sa-token" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188704 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="605b308f-b6a9-427f-a365-c0f8b110da64" volumeName="kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-stats-auth" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188715 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4cf2efa-495e-452f-b0a6-b17c1d54038b" volumeName="kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-trusted-ca" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188726 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7dfc7a1-6b86-471f-b85b-d5ea756db9cb" volumeName="kubernetes.io/configmap/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-telemetry-config" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188737 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d809989-435e-4d38-807e-4bdbe414cf32" volumeName="kubernetes.io/secret/1d809989-435e-4d38-807e-4bdbe414cf32-agent-certs" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188749 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70b0df95-c5a5-48b5-b670-8382137780ea" volumeName="kubernetes.io/configmap/70b0df95-c5a5-48b5-b670-8382137780ea-nginx-conf" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188760 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79411cba-ba1d-41d6-b656-a661befaaa11" volumeName="kubernetes.io/projected/79411cba-ba1d-41d6-b656-a661befaaa11-kube-api-access-9dsf9" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188771 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4cf2efa-495e-452f-b0a6-b17c1d54038b" volumeName="kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-kube-api-access-s5qrf" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188787 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8e59f23-4988-41f7-b462-912ac6c7c125" volumeName="kubernetes.io/secret/f8e59f23-4988-41f7-b462-912ac6c7c125-serving-cert" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188811 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feab0cd3-8408-4594-b808-95f5653188c2" volumeName="kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-etc-tuned" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188823 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8615b0-abab-426b-a0ee-893109807207" volumeName="kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-snapshots" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188836 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8615b0-abab-426b-a0ee-893109807207" volumeName="kubernetes.io/projected/2e8615b0-abab-426b-a0ee-893109807207-kube-api-access-wm957" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188847 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1697d46a-5b9e-429a-986f-76e472a662a5" volumeName="kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-registry-certificates" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188858 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2305471f-d5ac-43c7-b204-1650ae447ff9" volumeName="kubernetes.io/configmap/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-config" seLinuxMountContext="" Oct 30 19:20:39.188855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188871 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4cf2efa-495e-452f-b0a6-b17c1d54038b" volumeName="kubernetes.io/empty-dir/a4cf2efa-495e-452f-b0a6-b17c1d54038b-ca-trust-extracted" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188884 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2f5e9c8-28aa-4956-a3ee-811d5577ec5d" volumeName="kubernetes.io/projected/c2f5e9c8-28aa-4956-a3ee-811d5577ec5d-kube-api-access-zqt56" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188897 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc69b2d3-eac5-43d1-86e4-36fa7cda7618" volumeName="kubernetes.io/secret/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-serving-cert" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188907 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feab0cd3-8408-4594-b808-95f5653188c2" volumeName="kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-tmp" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188920 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1697d46a-5b9e-429a-986f-76e472a662a5" volumeName="kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-kube-api-access-lvkqc" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188931 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1697d46a-5b9e-429a-986f-76e472a662a5" volumeName="kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-image-registry-private-configuration" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188977 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="605b308f-b6a9-427f-a365-c0f8b110da64" volumeName="kubernetes.io/projected/605b308f-b6a9-427f-a365-c0f8b110da64-kube-api-access-2lkf8" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.188992 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4cf2efa-495e-452f-b0a6-b17c1d54038b" volumeName="kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-bound-sa-token" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189003 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4cf2efa-495e-452f-b0a6-b17c1d54038b" volumeName="kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-image-registry-private-configuration" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189026 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a94f5874-53f0-4be0-bb77-ae6d535356ef" volumeName="kubernetes.io/configmap/a94f5874-53f0-4be0-bb77-ae6d535356ef-serviceca" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189037 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc69b2d3-eac5-43d1-86e4-36fa7cda7618" volumeName="kubernetes.io/projected/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-kube-api-access-n8xmf" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189049 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f83325ae-1680-4213-94ca-12001250ffa9" volumeName="kubernetes.io/projected/f83325ae-1680-4213-94ca-12001250ffa9-kube-api-access-8md4c" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189060 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0926fd0c-2705-4d4d-afbd-7007b665bb4e" volumeName="kubernetes.io/configmap/0926fd0c-2705-4d4d-afbd-7007b665bb4e-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189071 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0926fd0c-2705-4d4d-afbd-7007b665bb4e" volumeName="kubernetes.io/projected/0926fd0c-2705-4d4d-afbd-7007b665bb4e-kube-api-access-tbt4q" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189083 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" volumeName="kubernetes.io/secret/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-serving-cert" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189094 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ac8874c-1de6-4a26-be0c-ad3a459c4d65" volumeName="kubernetes.io/projected/6ac8874c-1de6-4a26-be0c-ad3a459c4d65-kube-api-access-g2978" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189124 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e" volumeName="kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-config" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189136 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7dfc7a1-6b86-471f-b85b-d5ea756db9cb" volumeName="kubernetes.io/projected/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-kube-api-access-vpf8n" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189147 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f83325ae-1680-4213-94ca-12001250ffa9" volumeName="kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-multus-daemon-config" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189158 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f83325ae-1680-4213-94ca-12001250ffa9" volumeName="kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-cni-binary-copy" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189171 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8615b0-abab-426b-a0ee-893109807207" volumeName="kubernetes.io/secret/2e8615b0-abab-426b-a0ee-893109807207-serving-cert" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189182 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="044847a9-7c73-421b-98ae-50bbb505b30a" volumeName="kubernetes.io/configmap/044847a9-7c73-421b-98ae-50bbb505b30a-config-volume" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189193 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44b56555-103f-4cc0-8c1c-507c3cc48880" volumeName="kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189208 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44b56555-103f-4cc0-8c1c-507c3cc48880" volumeName="kubernetes.io/projected/44b56555-103f-4cc0-8c1c-507c3cc48880-kube-api-access-knvfn" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189219 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e" volumeName="kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189249 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e" volumeName="kubernetes.io/secret/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189261 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f" volumeName="kubernetes.io/projected/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kube-api-access-fhgjf" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189271 4904 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8e59f23-4988-41f7-b462-912ac6c7c125" volumeName="kubernetes.io/configmap/f8e59f23-4988-41f7-b462-912ac6c7c125-config" seLinuxMountContext="" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189285 4904 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:20:39.190797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.189292 4904 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:20:39.204871 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.204848 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/1.log" Oct 30 19:20:39.205899 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.205412 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/1.log" Oct 30 19:20:39.212499 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.212443 4904 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:20:39.214012 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.213992 4904 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:20:39.214012 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.214009 4904 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:20:39.214251 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.214025 4904 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:20:39.214251 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:39.214071 4904 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:20:39.224931 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.224909 4904 generic.go:356] "Generic (PLEG): container finished" podID="d4b92012d9bff88a0620cb1df1375601" containerID="365620a125dc1e8c6d51303685712212ef9ebae9055e3b253efa48772ac550b4" exitCode=0 Oct 30 19:20:39.235662 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.235601 4904 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:20:39.235662 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.235642 4904 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:20:39.237759 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.237724 4904 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:20:39.251236 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.251203 4904 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:20:39.251358 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.251301 4904 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:20:39.259421 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.259396 4904 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="348c838916d935d45e7d091be2e34a7df385ca4ab42816188cb56f227b749585" exitCode=0 Oct 30 19:20:39.259421 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.259421 4904 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="fda8c7f84c64ad867cd5ff2f000f7ef2b7ccd8e7313c2d889fd1ea7235128f08" exitCode=0 Oct 30 19:20:39.259554 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.259430 4904 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="0f03e039dd550729ef3dc994e58c8dd020973ececd24afe02458f0885767193f" exitCode=0 Oct 30 19:20:39.259554 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.259438 4904 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="c432066bc58a6c13a01a7870fc0208979114837b0fc498ffe10aa3cb5b7477f2" exitCode=0 Oct 30 19:20:39.259554 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.259447 4904 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="bef4fc9838e8c303159651e130642aca0378aeb232bde159048faf6668305de6" exitCode=0 Oct 30 19:20:39.259554 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.259452 4904 generic.go:356] "Generic (PLEG): container finished" podID="44b56555-103f-4cc0-8c1c-507c3cc48880" containerID="2c08f0884dc3cf6836b897c65d3d52797e9fb10eb995952e0121fa554689f80a" exitCode=0 Oct 30 19:20:39.266898 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.266882 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/1.log" Oct 30 19:20:39.266968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.266904 4904 generic.go:356] "Generic (PLEG): container finished" podID="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" containerID="b5d4347479a7c97c79847e088687098f7c161fa6231cab27c845f8a624842a4e" exitCode=255 Oct 30 19:20:39.275972 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.275959 4904 manager.go:324] Recovery completed Oct 30 19:20:39.296745 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.296722 4904 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:20:39.296745 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.296736 4904 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:20:39.296897 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.296755 4904 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:20:39.296968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.296898 4904 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:20:39.296968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.296912 4904 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:20:39.296968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.296937 4904 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:20:39.296968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.296946 4904 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:20:39.296968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.296954 4904 policy_none.go:50] "Start" Oct 30 19:20:39.296968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.296966 4904 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:20:39.297278 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.296979 4904 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:20:39.297278 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.297181 4904 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:20:39.297278 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.297192 4904 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:20:39.297278 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.297201 4904 policy_none.go:44] "Start" Oct 30 19:20:39.301461 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.301435 4904 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:20:39.304089 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.304067 4904 manager.go:341] "Starting Device Plugin manager" Oct 30 19:20:39.304618 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:39.304094 4904 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:20:39.304677 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.304641 4904 server.go:87] "Starting device plugin registration server" Oct 30 19:20:39.304882 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.304870 4904 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:20:39.304916 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.304884 4904 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:20:39.305003 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.304981 4904 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:20:39.305145 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.305090 4904 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:20:39.305223 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.305154 4904 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:20:39.305223 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.305162 4904 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:20:39.305326 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.305196 4904 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:20:39.305144196Z","UUID":"8eaabf46-33ed-46e1-aa48-9ea65dd1dabb","Handler":null,"Name":"","Endpoint":""} Oct 30 19:20:39.306517 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.306496 4904 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:20:39.306604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.306523 4904 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:20:39.306604 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:39.306536 4904 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:20:39.314181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.314141 4904 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal"] Oct 30 19:20:39.314797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.314769 4904 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.314896 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.314800 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" event={"ID":"d4b92012d9bff88a0620cb1df1375601","Type":"ContainerStarted","Data":"dea43ea506a7ac54ce6d954d86880737a0e5dccac675835bd8c89c72de01275e"} Oct 30 19:20:39.314896 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.314833 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" event={"ID":"d4b92012d9bff88a0620cb1df1375601","Type":"ContainerDied","Data":"365620a125dc1e8c6d51303685712212ef9ebae9055e3b253efa48772ac550b4"} Oct 30 19:20:39.314896 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.314854 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" event={"ID":"d4b92012d9bff88a0620cb1df1375601","Type":"ContainerStarted","Data":"650acbdd35e837e582c04501a2cbe8dac53e5203ca5d566f3503a94a2cf1bc46"} Oct 30 19:20:39.315062 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.314916 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" event={"ID":"62fcc3a86a753eb9d656b3c26d634e28","Type":"ContainerStarted","Data":"f29b70b4d2cbb236b2ebd3aeb3901a34e4c7ac48700a4239a66c9a377e44a9c3"} Oct 30 19:20:39.315062 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.314935 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" event={"ID":"62fcc3a86a753eb9d656b3c26d634e28","Type":"ContainerStarted","Data":"feb2c31699f11464ebd9050a87da5f7bf3c8d92fb4b9e9c849c6984486bb1696"} Oct 30 19:20:39.315221 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.315198 4904 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.345946 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.345921 4904 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:20:39.346093 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:39.345972 4904 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.347560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.347547 4904 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:20:39.347602 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:39.347581 4904 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.405300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.405270 4904 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:20:39.406569 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.406550 4904 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:20:39.406695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.406585 4904 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:20:39.406695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.406602 4904 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:20:39.406695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.406658 4904 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.436760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.436732 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62fcc3a86a753eb9d656b3c26d634e28-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" (UID: \"62fcc3a86a753eb9d656b3c26d634e28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.436876 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.436763 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.436876 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.436852 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.446073 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.446050 4904 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.446184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.446145 4904 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.537253 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.537172 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.537253 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.537208 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.537253 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.537232 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62fcc3a86a753eb9d656b3c26d634e28-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" (UID: \"62fcc3a86a753eb9d656b3c26d634e28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.537485 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.537282 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62fcc3a86a753eb9d656b3c26d634e28-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" (UID: \"62fcc3a86a753eb9d656b3c26d634e28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.537485 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.537289 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:20:39.537485 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:39.537306 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b92012d9bff88a0620cb1df1375601-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"d4b92012d9bff88a0620cb1df1375601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:20:40.170870 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.170837 4904 apiserver.go:52] "Watching apiserver" Oct 30 19:20:40.244375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.244346 4904 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:20:40.245034 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.245013 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk","openshift-network-diagnostics/network-check-target-zrrw2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m","openshift-ovn-kubernetes/ovnkube-node-f7zr5","kube-system/konnectivity-agent-ttpmm","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx","openshift-insights/insights-runtime-extractor-l55ql","openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m","kube-system/global-pull-secret-syncer-vfmk2","openshift-insights/insights-operator-7c4986d449-58fgj","openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q","openshift-image-registry/node-ca-jsp6k","openshift-dns/node-resolver-5lmmj","openshift-image-registry/image-registry-5cf497f4fc-s2vpv","openshift-ingress/router-default-56bc986487-ljfn2","openshift-multus/multus-h28qn","kube-system/kas-connection-checker-685895c7c9-gt86s","openshift-console-operator/console-operator-7cd56b8dbc-9g8dh","openshift-cluster-node-tuning-operator/tuned-cj445","openshift-multus/multus-additional-cni-plugins-8sm2r","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt","openshift-network-console/networking-console-plugin-86c478bb88-8ns8s","openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42","openshift-image-registry/image-registry-6d984dbd48-qljg6","openshift-ingress-canary/ingress-canary-hlqc9","kube-system/kas-connection-checker-685895c7c9-qgdzh","kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal","kube-system/kas-connection-checker-685895c7c9-c6bdc","openshift-multus/network-metrics-daemon-q7pb2","openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal","openshift-dns/dns-default-77k4g","openshift-network-operator/iptables-alerter-czfgl"] Oct 30 19:20:40.246709 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.246688 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:40.247698 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.247680 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.248003 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.247987 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:40.248647 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.248626 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.249411 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.249397 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:40.256548 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.256520 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:40.258327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.258310 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.259446 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.259431 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:40.260158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.260143 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.263315 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.263297 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 19:20:40.263403 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.263375 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 19:20:40.263473 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.263450 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vp447\"" Oct 30 19:20:40.263473 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.263462 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hh67w\"" Oct 30 19:20:40.271579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.271558 4904 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:20:40.271698 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.271686 4904 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:20:40.275532 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.275518 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zqwsx\"" Oct 30 19:20:40.277443 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.277428 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 19:20:40.277554 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.277540 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 19:20:40.277682 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.277671 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j2d4c\"" Oct 30 19:20:40.277796 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.277781 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zfxvd\"" Oct 30 19:20:40.277838 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.277832 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k56vl\"" Oct 30 19:20:40.277880 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.277863 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-65nsn\"" Oct 30 19:20:40.277911 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.277883 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-g6588\"" Oct 30 19:20:40.281852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.281835 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:20:40.281852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.281846 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:20:40.281969 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.281874 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bkrzr\"" Oct 30 19:20:40.282031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282014 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 19:20:40.282183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282162 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:20:40.282183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282174 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:20:40.282327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282200 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 19:20:40.282327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282221 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-w8vq8\"" Oct 30 19:20:40.282327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282240 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 19:20:40.282471 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282456 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-ww87v\"" Oct 30 19:20:40.282520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282504 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gx5x8\"" Oct 30 19:20:40.282686 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282672 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:20:40.282717 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282675 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m2qt2\"" Oct 30 19:20:40.282771 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282759 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 19:20:40.282821 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282791 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:20:40.282821 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282809 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:20:40.282821 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.282817 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hnk7h\"" Oct 30 19:20:40.285576 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.285562 4904 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:20:40.289154 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.289136 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b7kzg\"" Oct 30 19:20:40.289229 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.289222 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-77hs9\"" Oct 30 19:20:40.289269 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.289232 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lkdbb\"" Oct 30 19:20:40.290171 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290156 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290157 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290178 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290217 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4v55m\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290225 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qmnn8\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290221 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vp4s5\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290235 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gx26w\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290232 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bcv2g\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290250 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290232 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sl59z\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290257 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hxbtq\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290258 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5ffd5\"" Oct 30 19:20:40.290296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290277 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8rhbj\"" Oct 30 19:20:40.290897 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290860 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:20:40.290999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290910 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9smtf\"" Oct 30 19:20:40.290999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.290917 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:20:40.297384 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.297366 4904 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:20:40.297482 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.297408 4904 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" Oct 30 19:20:40.298316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.298299 4904 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:20:40.298414 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.298340 4904 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" Oct 30 19:20:40.328324 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.328295 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.329213 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.329189 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.329516 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.329495 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.329764 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.329741 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 19:20:40.329920 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.329822 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.329920 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.329872 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.330093 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.330077 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.330254 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.330231 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:20:40.330357 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.330332 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.330424 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.330272 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.330480 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.330472 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:20:40.330793 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.330774 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.331036 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.330868 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.331552 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.331529 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 19:20:40.331828 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.331811 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.331900 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.331834 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:20:40.331947 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.331927 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.331990 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.331965 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.332086 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332072 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.332248 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332231 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.332322 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332304 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.332375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332352 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.332535 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332519 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:20:40.332610 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332537 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 19:20:40.332610 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332568 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.332699 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332615 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.332699 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332663 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.332784 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332738 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:20:40.332835 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332790 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.332880 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332857 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.332935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332882 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.332935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.332925 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.333049 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333030 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.333134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333059 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 19:20:40.333193 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333137 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:20:40.333237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333187 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.333320 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333306 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:20:40.333404 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333374 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.333483 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333472 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.333523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333484 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.333674 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333657 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.333763 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333747 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:20:40.333984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333962 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.333984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333981 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 19:20:40.334125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.333964 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:20:40.334125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.334068 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.334211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.334126 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.334211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.334143 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 19:20:40.334729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.334709 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:20:40.334866 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.334849 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.334933 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.334921 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.335002 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.334992 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:20:40.335065 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.335054 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.335127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.335094 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:40.335202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.335187 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:40.335252 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.335208 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:20:40.335475 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.335460 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:20:40.341524 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341507 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-trusted-ca\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:40.341568 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341534 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.341568 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341553 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4scpw\" (UniqueName: \"kubernetes.io/projected/1741e885-17b0-42a2-abb0-92ae73771c59-kube-api-access-4scpw\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.341646 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341568 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-config\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.341646 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341592 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6p8r\" (UniqueName: \"kubernetes.io/projected/feab0cd3-8408-4594-b808-95f5653188c2-kube-api-access-p6p8r\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.341646 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341614 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-dbus\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:40.341789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341694 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1741e885-17b0-42a2-abb0-92ae73771c59-crio-socket\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.341789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341746 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdmf4\" (UniqueName: \"kubernetes.io/projected/285534af-06ec-4171-93d8-2e3b57e1f85b-kube-api-access-xdmf4\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:40.341789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341768 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm957\" (UniqueName: \"kubernetes.io/projected/2e8615b0-abab-426b-a0ee-893109807207-kube-api-access-wm957\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.341789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341783 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-systemd-units\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.341976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341800 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.341976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341825 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0926fd0c-2705-4d4d-afbd-7007b665bb4e-iptables-alerter-script\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:20:40.341976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341861 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-config\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.341976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341871 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2978\" (UniqueName: \"kubernetes.io/projected/6ac8874c-1de6-4a26-be0c-ad3a459c4d65-kube-api-access-g2978\") pod \"network-check-source-6f78d59f99-52vqt\" (UID: \"6ac8874c-1de6-4a26-be0c-ad3a459c4d65\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt" Oct 30 19:20:40.341976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341903 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70b0df95-c5a5-48b5-b670-8382137780ea-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:40.341976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341927 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-device-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.341976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341946 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-conf-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.341976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.341970 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-trusted-ca\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.342272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342005 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-conf\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.342272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342131 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-image-registry-private-configuration\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.342272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342150 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70b0df95-c5a5-48b5-b670-8382137780ea-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:40.342272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342234 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0926fd0c-2705-4d4d-afbd-7007b665bb4e-iptables-alerter-script\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:20:40.342272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342245 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-registry-certificates\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342275 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342294 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57kl\" (UniqueName: \"kubernetes.io/projected/a94f5874-53f0-4be0-bb77-ae6d535356ef-kube-api-access-j57kl\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342312 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc66h\" (UniqueName: \"kubernetes.io/projected/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-kube-api-access-gc66h\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342361 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4jx\" (UniqueName: \"kubernetes.io/projected/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-kube-api-access-gr4jx\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342360 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-image-registry-private-configuration\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342392 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkqc\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-kube-api-access-lvkqc\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342423 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkf8\" (UniqueName: \"kubernetes.io/projected/605b308f-b6a9-427f-a365-c0f8b110da64-kube-api-access-2lkf8\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342434 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-registry-certificates\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342449 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-multus-daemon-config\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342475 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-etc-kubernetes\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.342498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342502 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4cf2efa-495e-452f-b0a6-b17c1d54038b-ca-trust-extracted\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342528 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-system-cni-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342555 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342592 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342612 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4cf2efa-495e-452f-b0a6-b17c1d54038b-ca-trust-extracted\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342664 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-multus-certs\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342722 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-bound-sa-token\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342756 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342850 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-multus-daemon-config\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342877 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8md4c\" (UniqueName: \"kubernetes.io/projected/f83325ae-1680-4213-94ca-12001250ffa9-kube-api-access-8md4c\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342913 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342942 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.342968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.342970 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-serving-cert\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:40.343427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343122 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-certificates\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.343427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343147 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-serving-cert\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:40.343427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343242 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-etc-tuned\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.343427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343272 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d809989-435e-4d38-807e-4bdbe414cf32-konnectivity-ca\") pod \"konnectivity-agent-ttpmm\" (UID: \"1d809989-435e-4d38-807e-4bdbe414cf32\") " pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:20:40.343427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343278 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-certificates\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.343427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343299 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-socket-dir-parent\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.343427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343319 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-etc-tuned\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.343427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343361 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-var-lib-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.343427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343378 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.343427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343395 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2fc\" (UniqueName: \"kubernetes.io/projected/64a1dc94-4952-4113-b191-ca277a6a520a-kube-api-access-xl2fc\") pod \"migrator-66df5d496b-s9d5m\" (UID: \"64a1dc94-4952-4113-b191-ca277a6a520a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343463 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d809989-435e-4d38-807e-4bdbe414cf32-konnectivity-ca\") pod \"konnectivity-agent-ttpmm\" (UID: \"1d809989-435e-4d38-807e-4bdbe414cf32\") " pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343492 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-binary-copy\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343571 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-node-log\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343611 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-netd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343632 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56b5\" (UniqueName: \"kubernetes.io/projected/9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1-kube-api-access-t56b5\") pod \"kas-connection-checker-685895c7c9-qgdzh\" (UID: \"9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1\") " pod="kube-system/kas-connection-checker-685895c7c9-qgdzh" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343650 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmpf\" (UniqueName: \"kubernetes.io/projected/2305471f-d5ac-43c7-b204-1650ae447ff9-kube-api-access-pkmpf\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343665 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-multus\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343681 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-cnibin\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343701 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dsf9\" (UniqueName: \"kubernetes.io/projected/79411cba-ba1d-41d6-b656-a661befaaa11-kube-api-access-9dsf9\") pod \"ingress-canary-hlqc9\" (UID: \"79411cba-ba1d-41d6-b656-a661befaaa11\") " pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343748 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e8615b0-abab-426b-a0ee-893109807207-serving-cert\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.343827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.343778 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-binary-copy\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344053 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e8615b0-abab-426b-a0ee-893109807207-serving-cert\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344037 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-k8s-cni-cncf-io\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344132 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-slash\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344158 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-ovn\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344183 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344239 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344244 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-installation-pull-secrets\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344292 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79411cba-ba1d-41d6-b656-a661befaaa11-cert\") pod \"ingress-canary-hlqc9\" (UID: \"79411cba-ba1d-41d6-b656-a661befaaa11\") " pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344326 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/044847a9-7c73-421b-98ae-50bbb505b30a-config-volume\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344360 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-sys-fs\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.344396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344390 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhgjf\" (UniqueName: \"kubernetes.io/projected/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kube-api-access-fhgjf\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344421 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqlw\" (UniqueName: \"kubernetes.io/projected/b07766d9-9884-4f58-9c94-e59540953a06-kube-api-access-pdqlw\") pod \"kas-connection-checker-685895c7c9-c6bdc\" (UID: \"b07766d9-9884-4f58-9c94-e59540953a06\") " pod="kube-system/kas-connection-checker-685895c7c9-c6bdc" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344455 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-system-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344457 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-installation-pull-secrets\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344485 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-config\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344514 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-kubelet\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344544 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-sys\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344556 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344570 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344573 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-var-lib-kubelet\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344630 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79411cba-ba1d-41d6-b656-a661befaaa11-cert\") pod \"ingress-canary-hlqc9\" (UID: \"79411cba-ba1d-41d6-b656-a661befaaa11\") " pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344693 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbt4q\" (UniqueName: \"kubernetes.io/projected/0926fd0c-2705-4d4d-afbd-7007b665bb4e-kube-api-access-tbt4q\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344709 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344725 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8e59f23-4988-41f7-b462-912ac6c7c125-config\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344759 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-socket-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344794 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/044847a9-7c73-421b-98ae-50bbb505b30a-config-volume\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:40.344877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344788 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-kubernetes\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344911 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344972 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.344982 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8e59f23-4988-41f7-b462-912ac6c7c125-config\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345002 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-cni-binary-copy\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345027 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-bin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345075 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-bound-sa-token\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345142 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-etc-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345187 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f83325ae-1680-4213-94ca-12001250ffa9-cni-binary-copy\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345216 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knvfn\" (UniqueName: \"kubernetes.io/projected/44b56555-103f-4cc0-8c1c-507c3cc48880-kube-api-access-knvfn\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345269 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044847a9-7c73-421b-98ae-50bbb505b30a-tmp-dir\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345370 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-hostroot\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.345466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345402 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-image-registry-private-configuration\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.345856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345509 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044847a9-7c73-421b-98ae-50bbb505b30a-tmp-dir\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:40.345856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345551 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dqf\" (UniqueName: \"kubernetes.io/projected/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-kube-api-access-44dqf\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:40.345856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345562 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-image-registry-private-configuration\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.345856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345583 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-host\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.345856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345670 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:40.345856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345688 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-stats-auth\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.345856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345758 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.345856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345787 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:40.345856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345813 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-systemd\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.345856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345850 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8e59f23-4988-41f7-b462-912ac6c7c125-serving-cert\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345883 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-stats-auth\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345887 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6829a2b0-e64b-4b3c-9007-966545b88c8d-original-pull-secret\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345927 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktbxp\" (UniqueName: \"kubernetes.io/projected/044847a9-7c73-421b-98ae-50bbb505b30a-kube-api-access-ktbxp\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345956 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-tmp-dir\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.345984 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-kubelet-config\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346012 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-lib-modules\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346027 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8e59f23-4988-41f7-b462-912ac6c7c125-serving-cert\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346038 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346083 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-config\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346084 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:40.346162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346151 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-default-certificate\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346169 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-tmp-dir\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346169 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346175 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnlzn\" (UniqueName: \"kubernetes.io/projected/f8e59f23-4988-41f7-b462-912ac6c7c125-kube-api-access-fnlzn\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346221 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-snapshots\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346239 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346285 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346305 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-snapshots\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346307 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-trusted-ca\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346338 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346366 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346401 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-default-certificate\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346417 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-netns\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346435 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.346500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346465 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-kube-api-access-s5qrf\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346509 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-log-socket\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346600 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-bin\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346623 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-env-overrides\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346641 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-modprobe-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346719 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-service-ca-bundle\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346747 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346775 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-env-overrides\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346791 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a94f5874-53f0-4be0-bb77-ae6d535356ef-serviceca\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346840 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-systemd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346859 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-script-lib\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346884 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-service-ca-bundle\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346909 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.346989 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347008 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xmf\" (UniqueName: \"kubernetes.io/projected/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-kube-api-access-n8xmf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347053 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-installation-pull-secrets\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347065 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovnkube-script-lib\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347084 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpf8n\" (UniqueName: \"kubernetes.io/projected/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-kube-api-access-vpf8n\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347089 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-trusted-ca\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347121 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a94f5874-53f0-4be0-bb77-ae6d535356ef-serviceca\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347135 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d809989-435e-4d38-807e-4bdbe414cf32-agent-certs\") pod \"konnectivity-agent-ttpmm\" (UID: \"1d809989-435e-4d38-807e-4bdbe414cf32\") " pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347197 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e8615b0-abab-426b-a0ee-893109807207-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.347249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347252 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347267 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347278 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovn-node-metrics-cert\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347314 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-installation-pull-secrets\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347322 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1697d46a-5b9e-429a-986f-76e472a662a5-ca-trust-extracted\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347319 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d809989-435e-4d38-807e-4bdbe414cf32-agent-certs\") pod \"konnectivity-agent-ttpmm\" (UID: \"1d809989-435e-4d38-807e-4bdbe414cf32\") " pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347355 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1741e885-17b0-42a2-abb0-92ae73771c59-data-volume\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347385 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-tmp\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347390 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1697d46a-5b9e-429a-986f-76e472a662a5-ca-trust-extracted\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347438 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347459 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1741e885-17b0-42a2-abb0-92ae73771c59-data-volume\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347469 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347499 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-netns\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347515 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8615b0-abab-426b-a0ee-893109807207-tmp\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347549 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysconfig\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347573 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347646 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-ovn-node-metrics-cert\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347677 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44b56555-103f-4cc0-8c1c-507c3cc48880-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347698 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-registration-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347714 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-tmp\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347729 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0926fd0c-2705-4d4d-afbd-7007b665bb4e-host-slash\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347745 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347777 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-hosts-file\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347809 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feab0cd3-8408-4594-b808-95f5653188c2-tmp\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347865 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-cnibin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347899 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqt56\" (UniqueName: \"kubernetes.io/projected/c2f5e9c8-28aa-4956-a3ee-811d5577ec5d-kube-api-access-zqt56\") pod \"volume-data-source-validator-58c9f5f46d-fj5bt\" (UID: \"c2f5e9c8-28aa-4956-a3ee-811d5577ec5d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347928 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghlh\" (UniqueName: \"kubernetes.io/projected/e6811b53-7458-4b8c-b4f9-9f46fb029eea-kube-api-access-9ghlh\") pod \"kas-connection-checker-685895c7c9-gt86s\" (UID: \"e6811b53-7458-4b8c-b4f9-9f46fb029eea\") " pod="kube-system/kas-connection-checker-685895c7c9-gt86s" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347974 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a94f5874-53f0-4be0-bb77-ae6d535356ef-host\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347991 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.347997 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-run\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.348111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.348026 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-kubelet\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.352459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.352442 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-trusted-ca\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:40.352549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.352535 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-trusted-ca\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.364385 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.364361 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdmf4\" (UniqueName: \"kubernetes.io/projected/285534af-06ec-4171-93d8-2e3b57e1f85b-kube-api-access-xdmf4\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:40.364699 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.364684 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4scpw\" (UniqueName: \"kubernetes.io/projected/1741e885-17b0-42a2-abb0-92ae73771c59-kube-api-access-4scpw\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.364882 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.364865 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6p8r\" (UniqueName: \"kubernetes.io/projected/feab0cd3-8408-4594-b808-95f5653188c2-kube-api-access-p6p8r\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.365760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.365737 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2978\" (UniqueName: \"kubernetes.io/projected/6ac8874c-1de6-4a26-be0c-ad3a459c4d65-kube-api-access-g2978\") pod \"network-check-source-6f78d59f99-52vqt\" (UID: \"6ac8874c-1de6-4a26-be0c-ad3a459c4d65\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-52vqt" Oct 30 19:20:40.365919 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.365899 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm957\" (UniqueName: \"kubernetes.io/projected/2e8615b0-abab-426b-a0ee-893109807207-kube-api-access-wm957\") pod \"insights-operator-7c4986d449-58fgj\" (UID: \"2e8615b0-abab-426b-a0ee-893109807207\") " pod="openshift-insights/insights-operator-7c4986d449-58fgj" Oct 30 19:20:40.368469 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.368451 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqt56\" (UniqueName: \"kubernetes.io/projected/c2f5e9c8-28aa-4956-a3ee-811d5577ec5d-kube-api-access-zqt56\") pod \"volume-data-source-validator-58c9f5f46d-fj5bt\" (UID: \"c2f5e9c8-28aa-4956-a3ee-811d5577ec5d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fj5bt" Oct 30 19:20:40.369658 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.369638 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56b5\" (UniqueName: \"kubernetes.io/projected/9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1-kube-api-access-t56b5\") pod \"kas-connection-checker-685895c7c9-qgdzh\" (UID: \"9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1\") " pod="kube-system/kas-connection-checker-685895c7c9-qgdzh" Oct 30 19:20:40.369855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.369840 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-bound-sa-token\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.370447 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.370431 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktbxp\" (UniqueName: \"kubernetes.io/projected/044847a9-7c73-421b-98ae-50bbb505b30a-kube-api-access-ktbxp\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:40.370652 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.370636 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4jx\" (UniqueName: \"kubernetes.io/projected/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-kube-api-access-gr4jx\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:20:40.370709 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.370665 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmpf\" (UniqueName: \"kubernetes.io/projected/2305471f-d5ac-43c7-b204-1650ae447ff9-kube-api-access-pkmpf\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:40.371054 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.371035 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57kl\" (UniqueName: \"kubernetes.io/projected/a94f5874-53f0-4be0-bb77-ae6d535356ef-kube-api-access-j57kl\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:20:40.371784 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.371768 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-kube-api-access-s5qrf\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.372436 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.372413 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbt4q\" (UniqueName: \"kubernetes.io/projected/0926fd0c-2705-4d4d-afbd-7007b665bb4e-kube-api-access-tbt4q\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:20:40.372599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.372584 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc66h\" (UniqueName: \"kubernetes.io/projected/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-kube-api-access-gc66h\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.372717 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.372702 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhc7r\" (UniqueName: \"kubernetes.io/projected/625ee392-6184-45a1-83a4-f577ee2062a0-kube-api-access-mhc7r\") pod \"network-check-target-zrrw2\" (UID: \"625ee392-6184-45a1-83a4-f577ee2062a0\") " pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:40.372812 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.372795 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqlw\" (UniqueName: \"kubernetes.io/projected/b07766d9-9884-4f58-9c94-e59540953a06-kube-api-access-pdqlw\") pod \"kas-connection-checker-685895c7c9-c6bdc\" (UID: \"b07766d9-9884-4f58-9c94-e59540953a06\") " pod="kube-system/kas-connection-checker-685895c7c9-c6bdc" Oct 30 19:20:40.373208 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.373193 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2fc\" (UniqueName: \"kubernetes.io/projected/64a1dc94-4952-4113-b191-ca277a6a520a-kube-api-access-xl2fc\") pod \"migrator-66df5d496b-s9d5m\" (UID: \"64a1dc94-4952-4113-b191-ca277a6a520a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9d5m" Oct 30 19:20:40.373318 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.373304 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xmf\" (UniqueName: \"kubernetes.io/projected/dc69b2d3-eac5-43d1-86e4-36fa7cda7618-kube-api-access-n8xmf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk\" (UID: \"dc69b2d3-eac5-43d1-86e4-36fa7cda7618\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" Oct 30 19:20:40.374187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.374166 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dqf\" (UniqueName: \"kubernetes.io/projected/1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0-kube-api-access-44dqf\") pod \"console-operator-7cd56b8dbc-9g8dh\" (UID: \"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:40.374490 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.374470 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghlh\" (UniqueName: \"kubernetes.io/projected/e6811b53-7458-4b8c-b4f9-9f46fb029eea-kube-api-access-9ghlh\") pod \"kas-connection-checker-685895c7c9-gt86s\" (UID: \"e6811b53-7458-4b8c-b4f9-9f46fb029eea\") " pod="kube-system/kas-connection-checker-685895c7c9-gt86s" Oct 30 19:20:40.374802 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.374772 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8md4c\" (UniqueName: \"kubernetes.io/projected/f83325ae-1680-4213-94ca-12001250ffa9-kube-api-access-8md4c\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.374883 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.374775 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhgjf\" (UniqueName: \"kubernetes.io/projected/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kube-api-access-fhgjf\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.374937 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.374902 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpf8n\" (UniqueName: \"kubernetes.io/projected/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-kube-api-access-vpf8n\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:40.374937 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.374928 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkqc\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-kube-api-access-lvkqc\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.375006 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.374945 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knvfn\" (UniqueName: \"kubernetes.io/projected/44b56555-103f-4cc0-8c1c-507c3cc48880-kube-api-access-knvfn\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.375163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.375148 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnlzn\" (UniqueName: \"kubernetes.io/projected/f8e59f23-4988-41f7-b462-912ac6c7c125-kube-api-access-fnlzn\") pod \"service-ca-operator-544d749c8c-scc42\" (UID: \"f8e59f23-4988-41f7-b462-912ac6c7c125\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" Oct 30 19:20:40.375599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.375584 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkf8\" (UniqueName: \"kubernetes.io/projected/605b308f-b6a9-427f-a365-c0f8b110da64-kube-api-access-2lkf8\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.375921 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.375905 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dsf9\" (UniqueName: \"kubernetes.io/projected/79411cba-ba1d-41d6-b656-a661befaaa11-kube-api-access-9dsf9\") pod \"ingress-canary-hlqc9\" (UID: \"79411cba-ba1d-41d6-b656-a661befaaa11\") " pod="openshift-ingress-canary/ingress-canary-hlqc9" Oct 30 19:20:40.376322 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.376307 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-bound-sa-token\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.449404 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449328 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-log-socket\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.449404 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449359 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-bin\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.449404 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449376 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-modprobe-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.449404 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449398 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449420 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-systemd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449450 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-netns\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449455 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-log-socket\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449460 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-bin\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449493 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-netns\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449501 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-systemd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449521 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysconfig\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449509 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449561 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-modprobe-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449591 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysconfig\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449621 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-registration-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.449657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449647 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0926fd0c-2705-4d4d-afbd-7007b665bb4e-host-slash\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449672 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-hosts-file\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449676 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-registration-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449694 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-cnibin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449699 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0926fd0c-2705-4d4d-afbd-7007b665bb4e-host-slash\") pod \"iptables-alerter-czfgl\" (UID: \"0926fd0c-2705-4d4d-afbd-7007b665bb4e\") " pod="openshift-network-operator/iptables-alerter-czfgl" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449719 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a94f5874-53f0-4be0-bb77-ae6d535356ef-host\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449723 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d09042e-9b40-41d8-803d-8d4a0bbf0fd1-hosts-file\") pod \"node-resolver-5lmmj\" (UID: \"5d09042e-9b40-41d8-803d-8d4a0bbf0fd1\") " pod="openshift-dns/node-resolver-5lmmj" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449743 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-run\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449753 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a94f5874-53f0-4be0-bb77-ae6d535356ef-host\") pod \"node-ca-jsp6k\" (UID: \"a94f5874-53f0-4be0-bb77-ae6d535356ef\") " pod="openshift-image-registry/node-ca-jsp6k" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449763 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-kubelet\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449763 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-cnibin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449787 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449793 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-kubelet\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449805 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-run\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449805 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-dbus\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449840 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1741e885-17b0-42a2-abb0-92ae73771c59-crio-socket\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449847 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-dbus\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449863 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-systemd-units\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449879 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1741e885-17b0-42a2-abb0-92ae73771c59-crio-socket\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449882 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449905 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-run-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449881 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449916 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-systemd-units\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449900 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-device-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449951 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-device-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449962 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-conf-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.449988 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-conf\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450003 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-conf-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450015 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450044 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-etc-kubernetes\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450068 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-system-cni-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450089 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-conf\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450095 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450137 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-system-cni-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450139 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450147 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-etc-kubernetes\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450163 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-multus-certs\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450188 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450212 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-multus-certs\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450215 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.450235 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.95021949 +0000 UTC m=+1.863824098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : configmap references non-existent config key: service-ca.crt Oct 30 19:20:40.450217 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.450244 4904 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450250 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.450281 4904 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450295 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.450314 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs podName:285534af-06ec-4171-93d8-2e3b57e1f85b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.950297721 +0000 UTC m=+1.863902343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs") pod "network-metrics-daemon-q7pb2" (UID: "285534af-06ec-4171-93d8-2e3b57e1f85b") : secret "metrics-daemon-secret" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.450335 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls podName:044847a9-7c73-421b-98ae-50bbb505b30a nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.950325604 +0000 UTC m=+1.863930204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls") pod "dns-default-77k4g" (UID: "044847a9-7c73-421b-98ae-50bbb505b30a") : secret "dns-default-metrics-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.450361 4904 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450376 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-socket-dir-parent\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450402 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-var-lib-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.450412 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.950398018 +0000 UTC m=+1.864002628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : secret "router-metrics-certs-default" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450437 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-var-lib-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450439 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450460 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-multus-socket-dir-parent\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450473 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-node-log\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450497 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-netd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450525 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-multus\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450534 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-node-log\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450543 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-sysctl-d\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450548 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-cnibin\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450563 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-cni-netd\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450567 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-multus\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450583 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-k8s-cni-cncf-io\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450605 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-k8s-cni-cncf-io\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450608 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-slash\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450636 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1741e885-17b0-42a2-abb0-92ae73771c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450631 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-ovn\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450586 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-cnibin\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450659 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-slash\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450683 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450710 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-sys-fs\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450730 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-run-ovn\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450737 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-system-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450772 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-kubelet\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450784 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-sys-fs\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450789 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-system-cni-dir\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450772 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44b56555-103f-4cc0-8c1c-507c3cc48880-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8sm2r\" (UID: \"44b56555-103f-4cc0-8c1c-507c3cc48880\") " pod="openshift-multus/multus-additional-cni-plugins-8sm2r" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450803 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-host-kubelet\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450796 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-sys\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450831 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-sys\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450835 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-var-lib-kubelet\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450858 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-var-lib-kubelet\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450865 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-socket-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450881 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-kubernetes\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450908 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450927 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450942 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-bin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450950 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-socket-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450960 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-etc-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.450998 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-kubernetes\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451010 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e-etc-openvswitch\") pod \"ovnkube-node-f7zr5\" (UID: \"b9a4eb6e-f33f-4f60-9193-2dfbb3c2003e\") " pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451019 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-var-lib-cni-bin\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451031 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451053 4904 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451053 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-hostroot\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451088 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-hostroot\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451092 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls podName:2305471f-d5ac-43c7-b204-1650ae447ff9 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.951080303 +0000 UTC m=+1.864684910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dzhlx" (UID: "2305471f-d5ac-43c7-b204-1650ae447ff9") : secret "samples-operator-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451150 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-host\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451176 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451198 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-systemd\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451222 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-kubelet-config\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451242 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-lib-modules\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451265 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451288 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451315 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451333 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-host\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451341 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451318 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-etc-systemd\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451366 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-netns\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451369 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6829a2b0-e64b-4b3c-9007-966545b88c8d-kubelet-config\") pod \"global-pull-secret-syncer-vfmk2\" (UID: \"6829a2b0-e64b-4b3c-9007-966545b88c8d\") " pod="kube-system/global-pull-secret-syncer-vfmk2" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451392 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451408 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f83325ae-1680-4213-94ca-12001250ffa9-host-run-netns\") pod \"multus-h28qn\" (UID: \"f83325ae-1680-4213-94ca-12001250ffa9\") " pod="openshift-multus/multus-h28qn" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451367 4904 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451408 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7t6m\" (UID: \"c8a0dec1-fbc1-477b-9aaa-b921fc3ff46f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7t6m" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451403 4904 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451421 4904 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451440 4904 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf497f4fc-s2vpv: secret "image-registry-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451409 4904 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.451486 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feab0cd3-8408-4594-b808-95f5653188c2-lib-modules\") pod \"tuned-cj445\" (UID: \"feab0cd3-8408-4594-b808-95f5653188c2\") " pod="openshift-cluster-node-tuning-operator/tuned-cj445" Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451492 4904 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451518 4904 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d984dbd48-qljg6: secret "image-registry-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451491 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls podName:1697d46a-5b9e-429a-986f-76e472a662a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.951476621 +0000 UTC m=+1.865081233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls") pod "image-registry-5cf497f4fc-s2vpv" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5") : secret "image-registry-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451545 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls podName:1741e885-17b0-42a2-abb0-92ae73771c59 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.951534089 +0000 UTC m=+1.865138687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l55ql" (UID: "1741e885-17b0-42a2-abb0-92ae73771c59") : secret "insights-runtime-extractor-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451563 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls podName:d7dfc7a1-6b86-471f-b85b-d5ea756db9cb nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.951553963 +0000 UTC m=+1.865158565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fgx9q" (UID: "d7dfc7a1-6b86-471f-b85b-d5ea756db9cb") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451578 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert podName:70b0df95-c5a5-48b5-b670-8382137780ea nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.951572239 +0000 UTC m=+1.865176834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8ns8s" (UID: "70b0df95-c5a5-48b5-b670-8382137780ea") : secret "networking-console-plugin-cert" not found Oct 30 19:20:40.451852 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.451587 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls podName:a4cf2efa-495e-452f-b0a6-b17c1d54038b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.951582354 +0000 UTC m=+1.865186950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls") pod "image-registry-6d984dbd48-qljg6" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b") : secret "image-registry-tls" not found Oct 30 19:20:40.548745 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.548715 4904 scope.go:122] "RemoveContainer" containerID="b5d4347479a7c97c79847e088687098f7c161fa6231cab27c845f8a624842a4e" Oct 30 19:20:40.945751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.945684 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:40.955071 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.955050 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:40.955184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.955082 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:40.955184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.955126 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:40.955184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.955148 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:40.955184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.955164 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:40.955184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.955185 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:40.955410 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955196 4904 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:20:40.955410 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.955243 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.955410 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955246 4904 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:40.955410 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955253 4904 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:20:40.955410 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955282 4904 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:40.955410 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955245 4904 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:20:40.955410 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955301 4904 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf497f4fc-s2vpv: secret "image-registry-tls" not found Oct 30 19:20:40.955410 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955308 4904 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:40.955410 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955325 4904 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d984dbd48-qljg6: secret "image-registry-tls" not found Oct 30 19:20:40.955410 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955263 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls podName:2305471f-d5ac-43c7-b204-1650ae447ff9 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:41.955244343 +0000 UTC m=+2.868848952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dzhlx" (UID: "2305471f-d5ac-43c7-b204-1650ae447ff9") : secret "samples-operator-tls" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955431 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert podName:70b0df95-c5a5-48b5-b670-8382137780ea nodeName:}" failed. No retries permitted until 2026-10-30 19:20:41.95541865 +0000 UTC m=+2.869023246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8ns8s" (UID: "70b0df95-c5a5-48b5-b670-8382137780ea") : secret "networking-console-plugin-cert" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955447 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:41.955440739 +0000 UTC m=+2.869045334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : configmap references non-existent config key: service-ca.crt Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955460 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls podName:1741e885-17b0-42a2-abb0-92ae73771c59 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:41.955452443 +0000 UTC m=+2.869057043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l55ql" (UID: "1741e885-17b0-42a2-abb0-92ae73771c59") : secret "insights-runtime-extractor-tls" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955475 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls podName:a4cf2efa-495e-452f-b0a6-b17c1d54038b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:41.955467215 +0000 UTC m=+2.869071836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls") pod "image-registry-6d984dbd48-qljg6" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b") : secret "image-registry-tls" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955495 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls podName:d7dfc7a1-6b86-471f-b85b-d5ea756db9cb nodeName:}" failed. No retries permitted until 2026-10-30 19:20:41.955487503 +0000 UTC m=+2.869092098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fgx9q" (UID: "d7dfc7a1-6b86-471f-b85b-d5ea756db9cb") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955512 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls podName:1697d46a-5b9e-429a-986f-76e472a662a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:41.955504253 +0000 UTC m=+2.869108851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls") pod "image-registry-5cf497f4fc-s2vpv" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5") : secret "image-registry-tls" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.955541 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.955565 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:40.955584 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955677 4904 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955684 4904 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955717 4904 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955720 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs podName:285534af-06ec-4171-93d8-2e3b57e1f85b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:41.955707509 +0000 UTC m=+2.869312118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs") pod "network-metrics-daemon-q7pb2" (UID: "285534af-06ec-4171-93d8-2e3b57e1f85b") : secret "metrics-daemon-secret" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955807 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:41.95578891 +0000 UTC m=+2.869393513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : secret "router-metrics-certs-default" not found Oct 30 19:20:40.955849 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:40.955825 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls podName:044847a9-7c73-421b-98ae-50bbb505b30a nodeName:}" failed. No retries permitted until 2026-10-30 19:20:41.955815317 +0000 UTC m=+2.869419930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls") pod "dns-default-77k4g" (UID: "044847a9-7c73-421b-98ae-50bbb505b30a") : secret "dns-default-metrics-tls" not found Oct 30 19:20:41.275891 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.275816 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:20:41.276296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.276250 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/1.log" Oct 30 19:20:41.276335 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.276292 4904 generic.go:356] "Generic (PLEG): container finished" podID="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" containerID="585ebd014c98283cae829db3b31d64f870ce7b106f0ec5b7e89052cae4d2b6f8" exitCode=255 Oct 30 19:20:41.276419 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.276393 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" event={"ID":"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0","Type":"ContainerDied","Data":"585ebd014c98283cae829db3b31d64f870ce7b106f0ec5b7e89052cae4d2b6f8"} Oct 30 19:20:41.276471 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.276458 4904 scope.go:122] "RemoveContainer" containerID="b5d4347479a7c97c79847e088687098f7c161fa6231cab27c845f8a624842a4e" Oct 30 19:20:41.276549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.276537 4904 scope.go:122] "RemoveContainer" containerID="585ebd014c98283cae829db3b31d64f870ce7b106f0ec5b7e89052cae4d2b6f8" Oct 30 19:20:41.276747 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.276725 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9g8dh_openshift-console-operator(1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" podUID="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" Oct 30 19:20:41.602564 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.602473 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:41.964874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.964789 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:41.964874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.964834 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:41.964874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.964855 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:41.964874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.964873 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.964891 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.964920 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.964951 4904 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.964974 4904 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.964989 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965015 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert podName:70b0df95-c5a5-48b5-b670-8382137780ea nodeName:}" failed. No retries permitted until 2026-10-30 19:20:43.965001122 +0000 UTC m=+4.878605718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8ns8s" (UID: "70b0df95-c5a5-48b5-b670-8382137780ea") : secret "networking-console-plugin-cert" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.964951 4904 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965021 4904 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965047 4904 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf497f4fc-s2vpv: secret "image-registry-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.964950 4904 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965062 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls podName:2305471f-d5ac-43c7-b204-1650ae447ff9 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:43.965048588 +0000 UTC m=+4.878653195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dzhlx" (UID: "2305471f-d5ac-43c7-b204-1650ae447ff9") : secret "samples-operator-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965061 4904 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965079 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls podName:d7dfc7a1-6b86-471f-b85b-d5ea756db9cb nodeName:}" failed. No retries permitted until 2026-10-30 19:20:43.965070239 +0000 UTC m=+4.878674835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fgx9q" (UID: "d7dfc7a1-6b86-471f-b85b-d5ea756db9cb") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965085 4904 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d984dbd48-qljg6: secret "image-registry-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965094 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:43.965086401 +0000 UTC m=+4.878690996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : configmap references non-existent config key: service-ca.crt Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.965043 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965137 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls podName:1697d46a-5b9e-429a-986f-76e472a662a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:43.965126472 +0000 UTC m=+4.878731069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls") pod "image-registry-5cf497f4fc-s2vpv" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5") : secret "image-registry-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965165 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls podName:1741e885-17b0-42a2-abb0-92ae73771c59 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:43.965148659 +0000 UTC m=+4.878753256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l55ql" (UID: "1741e885-17b0-42a2-abb0-92ae73771c59") : secret "insights-runtime-extractor-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965184 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls podName:a4cf2efa-495e-452f-b0a6-b17c1d54038b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:43.965176437 +0000 UTC m=+4.878781032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls") pod "image-registry-6d984dbd48-qljg6" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b") : secret "image-registry-tls" not found Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.965205 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:41.965211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.965225 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:41.965928 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965301 4904 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:20:41.965928 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:41.965335 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:43.965324535 +0000 UTC m=+4.878929130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : secret "router-metrics-certs-default" not found Oct 30 19:20:41.965928 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.965411 4904 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:20:41.967570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.967552 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285534af-06ec-4171-93d8-2e3b57e1f85b-metrics-certs\") pod \"network-metrics-daemon-q7pb2\" (UID: \"285534af-06ec-4171-93d8-2e3b57e1f85b\") " pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:41.967666 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:41.967651 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/044847a9-7c73-421b-98ae-50bbb505b30a-metrics-tls\") pod \"dns-default-77k4g\" (UID: \"044847a9-7c73-421b-98ae-50bbb505b30a\") " pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:42.057597 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:42.057561 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7pb2" Oct 30 19:20:42.060180 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:42.060161 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:42.283925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:42.283847 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:20:42.284305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:42.284186 4904 scope.go:122] "RemoveContainer" containerID="585ebd014c98283cae829db3b31d64f870ce7b106f0ec5b7e89052cae4d2b6f8" Oct 30 19:20:42.284385 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:42.284365 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9g8dh_openshift-console-operator(1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" podUID="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" Oct 30 19:20:42.387731 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:42.387700 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q7pb2"] Oct 30 19:20:42.391528 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:42.391494 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285534af_06ec_4171_93d8_2e3b57e1f85b.slice/crio-6ec3206f5af301d9efdf75414a33c715c8bc0267755101d521a25a959a5cdf71 WatchSource:0}: Error finding container 6ec3206f5af301d9efdf75414a33c715c8bc0267755101d521a25a959a5cdf71: Status 404 returned error can't find the container with id 6ec3206f5af301d9efdf75414a33c715c8bc0267755101d521a25a959a5cdf71 Oct 30 19:20:42.394236 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:42.394220 4904 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:20:42.398215 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:42.398193 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-77k4g"] Oct 30 19:20:42.401665 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:42.401643 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044847a9_7c73_421b_98ae_50bbb505b30a.slice/crio-55638dc985e5d5e37bbdd987f2c6062361d5700e915622c0b6ea995a2da1bcab WatchSource:0}: Error finding container 55638dc985e5d5e37bbdd987f2c6062361d5700e915622c0b6ea995a2da1bcab: Status 404 returned error can't find the container with id 55638dc985e5d5e37bbdd987f2c6062361d5700e915622c0b6ea995a2da1bcab Oct 30 19:20:43.126711 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.126677 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5lmmj_5d09042e-9b40-41d8-803d-8d4a0bbf0fd1/dns-node-resolver/0.log" Oct 30 19:20:43.290902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.290872 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-77k4g" event={"ID":"044847a9-7c73-421b-98ae-50bbb505b30a","Type":"ContainerStarted","Data":"55638dc985e5d5e37bbdd987f2c6062361d5700e915622c0b6ea995a2da1bcab"} Oct 30 19:20:43.292421 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.292394 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7pb2" event={"ID":"285534af-06ec-4171-93d8-2e3b57e1f85b","Type":"ContainerStarted","Data":"6ec3206f5af301d9efdf75414a33c715c8bc0267755101d521a25a959a5cdf71"} Oct 30 19:20:43.984843 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.984807 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:43.984992 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.984856 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:43.984992 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.984890 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:43.984992 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.984921 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:43.984992 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.984947 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:43.984992 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.984960 4904 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:20:43.985207 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.985008 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:43.985207 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985016 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls podName:2305471f-d5ac-43c7-b204-1650ae447ff9 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:47.985003363 +0000 UTC m=+8.898607958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dzhlx" (UID: "2305471f-d5ac-43c7-b204-1650ae447ff9") : secret "samples-operator-tls" not found Oct 30 19:20:43.985207 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985042 4904 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:43.985207 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985095 4904 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:20:43.985207 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985122 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls podName:d7dfc7a1-6b86-471f-b85b-d5ea756db9cb nodeName:}" failed. No retries permitted until 2026-10-30 19:20:47.985085477 +0000 UTC m=+8.898690092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fgx9q" (UID: "d7dfc7a1-6b86-471f-b85b-d5ea756db9cb") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:20:43.985207 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985153 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert podName:70b0df95-c5a5-48b5-b670-8382137780ea nodeName:}" failed. No retries permitted until 2026-10-30 19:20:47.985140844 +0000 UTC m=+8.898745448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8ns8s" (UID: "70b0df95-c5a5-48b5-b670-8382137780ea") : secret "networking-console-plugin-cert" not found Oct 30 19:20:43.985207 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985175 4904 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:43.985207 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985187 4904 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d984dbd48-qljg6: secret "image-registry-tls" not found Oct 30 19:20:43.985207 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985196 4904 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:20:43.985207 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985208 4904 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cf497f4fc-s2vpv: secret "image-registry-tls" not found Oct 30 19:20:43.985517 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985226 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls podName:a4cf2efa-495e-452f-b0a6-b17c1d54038b nodeName:}" failed. No retries permitted until 2026-10-30 19:20:47.985213555 +0000 UTC m=+8.898818154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls") pod "image-registry-6d984dbd48-qljg6" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b") : secret "image-registry-tls" not found Oct 30 19:20:43.985517 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.985183 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:43.985517 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985242 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:47.985233318 +0000 UTC m=+8.898837931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : configmap references non-existent config key: service-ca.crt Oct 30 19:20:43.985517 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:43.985263 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls podName:1697d46a-5b9e-429a-986f-76e472a662a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:47.985254368 +0000 UTC m=+8.898858986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls") pod "image-registry-5cf497f4fc-s2vpv" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5") : secret "image-registry-tls" not found Oct 30 19:20:43.985517 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.985288 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:43.987390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.987370 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1741e885-17b0-42a2-abb0-92ae73771c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l55ql\" (UID: \"1741e885-17b0-42a2-abb0-92ae73771c59\") " pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:43.987957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:43.987938 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605b308f-b6a9-427f-a365-c0f8b110da64-metrics-certs\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:44.126337 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:44.126301 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jsp6k_a94f5874-53f0-4be0-bb77-ae6d535356ef/node-ca/0.log" Oct 30 19:20:44.161031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:44.161008 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l55ql" Oct 30 19:20:44.291158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:44.291085 4904 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:44.291479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:44.291465 4904 scope.go:122] "RemoveContainer" containerID="585ebd014c98283cae829db3b31d64f870ce7b106f0ec5b7e89052cae4d2b6f8" Oct 30 19:20:44.291636 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:44.291617 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9g8dh_openshift-console-operator(1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" podUID="1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0" Oct 30 19:20:44.296898 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:44.296867 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-77k4g" event={"ID":"044847a9-7c73-421b-98ae-50bbb505b30a","Type":"ContainerStarted","Data":"00fd1e788653d3ad0a3c002c50a00c649c2e6a133c31b9b68c686a3dc7e2c045"} Oct 30 19:20:44.297007 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:44.296922 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-77k4g" event={"ID":"044847a9-7c73-421b-98ae-50bbb505b30a","Type":"ContainerStarted","Data":"eae0721e29b80a8aeea5e3a5bd93a655db77ced9139bbc1cc999c84f88e7f3a7"} Oct 30 19:20:44.297007 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:44.296971 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:44.298724 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:44.298705 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7pb2" event={"ID":"285534af-06ec-4171-93d8-2e3b57e1f85b","Type":"ContainerStarted","Data":"982ef83877085fd8b7d9f0a53872984f6bfa59e01c529635058156eeae065c5f"} Oct 30 19:20:44.298817 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:44.298749 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7pb2" event={"ID":"285534af-06ec-4171-93d8-2e3b57e1f85b","Type":"ContainerStarted","Data":"2af843c261315e19172acac68dd57d6bf7e6a019908188fb51041dcde35fa794"} Oct 30 19:20:44.460331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:44.460309 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l55ql"] Oct 30 19:20:44.464664 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:44.464632 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1741e885_17b0_42a2_abb0_92ae73771c59.slice/crio-cc4389f8c40b8a4191de3453ab303570e9885d52fb2abaf7eed59ecddfe8e6a9 WatchSource:0}: Error finding container cc4389f8c40b8a4191de3453ab303570e9885d52fb2abaf7eed59ecddfe8e6a9: Status 404 returned error can't find the container with id cc4389f8c40b8a4191de3453ab303570e9885d52fb2abaf7eed59ecddfe8e6a9 Oct 30 19:20:45.126396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:45.126374 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hlqc9_79411cba-ba1d-41d6-b656-a661befaaa11/serve-healthcheck-canary/0.log" Oct 30 19:20:45.303963 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:45.303928 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l55ql" event={"ID":"1741e885-17b0-42a2-abb0-92ae73771c59","Type":"ContainerStarted","Data":"733ec6f67ead0f2b66176cfafd10237fd276d79fa98c7d04be51cbf796b8ffc4"} Oct 30 19:20:45.304299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:45.303984 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l55ql" event={"ID":"1741e885-17b0-42a2-abb0-92ae73771c59","Type":"ContainerStarted","Data":"c1bef4860ab5f619c999f1542b1a85cf424d01638a44bd1619bcfce9dd270599"} Oct 30 19:20:45.304299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:45.304003 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l55ql" event={"ID":"1741e885-17b0-42a2-abb0-92ae73771c59","Type":"ContainerStarted","Data":"cc4389f8c40b8a4191de3453ab303570e9885d52fb2abaf7eed59ecddfe8e6a9"} Oct 30 19:20:45.526987 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:45.526954 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-s9d5m_64a1dc94-4952-4113-b191-ca277a6a520a/migrator/0.log" Oct 30 19:20:45.728144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:45.728120 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk_dc69b2d3-eac5-43d1-86e4-36fa7cda7618/kube-storage-version-migrator-operator/0.log" Oct 30 19:20:47.315720 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:47.315685 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l55ql" event={"ID":"1741e885-17b0-42a2-abb0-92ae73771c59","Type":"ContainerStarted","Data":"18986abf21090336726eda20711bc2073618b83aca1bde19f3f4025137c95cc8"} Oct 30 19:20:48.024059 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.024023 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:48.024265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.024074 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:48.024265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.024124 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:48.024265 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:48.024205 4904 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:20:48.024265 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:48.024212 4904 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:20:48.024265 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:48.024228 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle podName:605b308f-b6a9-427f-a365-c0f8b110da64 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:56.024206754 +0000 UTC m=+16.937811409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle") pod "router-default-56bc986487-ljfn2" (UID: "605b308f-b6a9-427f-a365-c0f8b110da64") : configmap references non-existent config key: service-ca.crt Oct 30 19:20:48.024265 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:48.024259 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert podName:70b0df95-c5a5-48b5-b670-8382137780ea nodeName:}" failed. No retries permitted until 2026-10-30 19:20:56.024246911 +0000 UTC m=+16.937851519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8ns8s" (UID: "70b0df95-c5a5-48b5-b670-8382137780ea") : secret "networking-console-plugin-cert" not found Oct 30 19:20:48.024600 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:48.024286 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls podName:2305471f-d5ac-43c7-b204-1650ae447ff9 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:56.024267155 +0000 UTC m=+16.937871749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dzhlx" (UID: "2305471f-d5ac-43c7-b204-1650ae447ff9") : secret "samples-operator-tls" not found Oct 30 19:20:48.024600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.024307 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:48.024600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.024332 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:48.024600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.024350 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:48.027337 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.027309 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"image-registry-6d984dbd48-qljg6\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:48.027471 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.027335 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"image-registry-5cf497f4fc-s2vpv\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:48.027471 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.027346 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7dfc7a1-6b86-471f-b85b-d5ea756db9cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fgx9q\" (UID: \"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:48.048450 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.048430 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" Oct 30 19:20:48.049569 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.049524 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:48.059503 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.059485 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:48.265544 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.265517 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ttpmm" Oct 30 19:20:48.289780 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.289722 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:20:48.289915 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.289904 4904 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:20:48.384499 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.384439 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q"] Oct 30 19:20:48.388520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.388495 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf497f4fc-s2vpv"] Oct 30 19:20:48.388654 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:48.388601 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7dfc7a1_6b86_471f_b85b_d5ea756db9cb.slice/crio-6a024446346cae6ec6b37f36ed0f6ba827f06cd164b04a0b810608ef6863fa65 WatchSource:0}: Error finding container 6a024446346cae6ec6b37f36ed0f6ba827f06cd164b04a0b810608ef6863fa65: Status 404 returned error can't find the container with id 6a024446346cae6ec6b37f36ed0f6ba827f06cd164b04a0b810608ef6863fa65 Oct 30 19:20:48.391574 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:48.391552 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1697d46a_5b9e_429a_986f_76e472a662a5.slice/crio-98049724bbfc165f04e14d5c12416dbf85a0bdb56ea24070ffe2fe799ba2e1fa WatchSource:0}: Error finding container 98049724bbfc165f04e14d5c12416dbf85a0bdb56ea24070ffe2fe799ba2e1fa: Status 404 returned error can't find the container with id 98049724bbfc165f04e14d5c12416dbf85a0bdb56ea24070ffe2fe799ba2e1fa Oct 30 19:20:48.400219 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:48.400195 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d984dbd48-qljg6"] Oct 30 19:20:48.404458 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:48.404433 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4cf2efa_495e_452f_b0a6_b17c1d54038b.slice/crio-d295bf5871fc79f706a19478910dc30e18e9175e2cce49f9350ee37deb999357 WatchSource:0}: Error finding container d295bf5871fc79f706a19478910dc30e18e9175e2cce49f9350ee37deb999357: Status 404 returned error can't find the container with id d295bf5871fc79f706a19478910dc30e18e9175e2cce49f9350ee37deb999357 Oct 30 19:20:49.034344 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:49.034311 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:20:49.324010 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:49.323931 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" event={"ID":"a4cf2efa-495e-452f-b0a6-b17c1d54038b","Type":"ContainerStarted","Data":"52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77"} Oct 30 19:20:49.324010 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:49.323982 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" event={"ID":"a4cf2efa-495e-452f-b0a6-b17c1d54038b","Type":"ContainerStarted","Data":"d295bf5871fc79f706a19478910dc30e18e9175e2cce49f9350ee37deb999357"} Oct 30 19:20:49.325360 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:49.325329 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" event={"ID":"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb","Type":"ContainerStarted","Data":"6a024446346cae6ec6b37f36ed0f6ba827f06cd164b04a0b810608ef6863fa65"} Oct 30 19:20:49.326833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:49.326809 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" event={"ID":"1697d46a-5b9e-429a-986f-76e472a662a5","Type":"ContainerStarted","Data":"57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87"} Oct 30 19:20:49.326950 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:49.326846 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" event={"ID":"1697d46a-5b9e-429a-986f-76e472a662a5","Type":"ContainerStarted","Data":"98049724bbfc165f04e14d5c12416dbf85a0bdb56ea24070ffe2fe799ba2e1fa"} Oct 30 19:20:50.331380 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:50.331344 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fgx9q" event={"ID":"d7dfc7a1-6b86-471f-b85b-d5ea756db9cb","Type":"ContainerStarted","Data":"6cfed3fb073ea132ee3ee51af703cd746eace6e5a406d60206beccd4471fa39c"} Oct 30 19:20:50.331380 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:50.331375 4904 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:20:50.331782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:50.331407 4904 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:20:50.946229 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:50.946190 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:50.946576 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:50.946561 4904 scope.go:122] "RemoveContainer" containerID="585ebd014c98283cae829db3b31d64f870ce7b106f0ec5b7e89052cae4d2b6f8" Oct 30 19:20:51.336324 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:51.336256 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:20:51.336782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:51.336382 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" event={"ID":"1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0","Type":"ContainerStarted","Data":"d8f6c8de3ac0884110c9010c0abe1a0547f551e49f78b8b422eb518c65f6a0a7"} Oct 30 19:20:51.336782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:51.336697 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:51.341622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:51.341601 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9g8dh" Oct 30 19:20:53.750839 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:53.750808 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:20:54.305762 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:54.305735 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-77k4g" Oct 30 19:20:54.506889 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:20:54.506858 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:20:56.093718 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.093677 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:56.094161 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.093732 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:56.094161 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.093777 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:56.094496 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.094465 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605b308f-b6a9-427f-a365-c0f8b110da64-service-ca-bundle\") pod \"router-default-56bc986487-ljfn2\" (UID: \"605b308f-b6a9-427f-a365-c0f8b110da64\") " pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:56.096858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.096827 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70b0df95-c5a5-48b5-b670-8382137780ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8ns8s\" (UID: \"70b0df95-c5a5-48b5-b670-8382137780ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:56.096991 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.096909 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2305471f-d5ac-43c7-b204-1650ae447ff9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dzhlx\" (UID: \"2305471f-d5ac-43c7-b204-1650ae447ff9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:56.147723 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.147697 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" Oct 30 19:20:56.148811 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.148793 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:56.150927 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.150908 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" Oct 30 19:20:56.271276 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.271248 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:20:56.461744 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.461718 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx"] Oct 30 19:20:56.470518 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:56.470492 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b0df95_c5a5_48b5_b670_8382137780ea.slice/crio-aa0741f669075d6b1b3372f2405c62a2ec060b18c548974dae79df88dc37deed WatchSource:0}: Error finding container aa0741f669075d6b1b3372f2405c62a2ec060b18c548974dae79df88dc37deed: Status 404 returned error can't find the container with id aa0741f669075d6b1b3372f2405c62a2ec060b18c548974dae79df88dc37deed Oct 30 19:20:56.475253 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.475231 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8ns8s"] Oct 30 19:20:56.477330 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:56.477308 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56bc986487-ljfn2"] Oct 30 19:20:56.480956 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:20:56.480931 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605b308f_b6a9_427f_a365_c0f8b110da64.slice/crio-7f3057839045e0e9f207921594801dcb687f3353412c7f2533f3596232ee811d WatchSource:0}: Error finding container 7f3057839045e0e9f207921594801dcb687f3353412c7f2533f3596232ee811d: Status 404 returned error can't find the container with id 7f3057839045e0e9f207921594801dcb687f3353412c7f2533f3596232ee811d Oct 30 19:20:57.358437 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:57.358396 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" event={"ID":"2305471f-d5ac-43c7-b204-1650ae447ff9","Type":"ContainerStarted","Data":"c7206223f5756d76947c5b434171fa7c34257b55fc32a34899454805d351d3e7"} Oct 30 19:20:57.359987 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:57.359958 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" event={"ID":"70b0df95-c5a5-48b5-b670-8382137780ea","Type":"ContainerStarted","Data":"aa0741f669075d6b1b3372f2405c62a2ec060b18c548974dae79df88dc37deed"} Oct 30 19:20:57.361772 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:57.361748 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56bc986487-ljfn2" event={"ID":"605b308f-b6a9-427f-a365-c0f8b110da64","Type":"ContainerStarted","Data":"68b439ef3f2e2db4884d0e370608603e1d7847630a69bd0de60f2421e2ef1340"} Oct 30 19:20:57.361882 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:57.361792 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56bc986487-ljfn2" event={"ID":"605b308f-b6a9-427f-a365-c0f8b110da64","Type":"ContainerStarted","Data":"7f3057839045e0e9f207921594801dcb687f3353412c7f2533f3596232ee811d"} Oct 30 19:20:58.149247 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:58.149229 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:58.151729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:58.151712 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:58.367749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:58.367680 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" event={"ID":"2305471f-d5ac-43c7-b204-1650ae447ff9","Type":"ContainerStarted","Data":"19490369ecc1909c7b2b7484e5fba1ea120034470c58625d81732358d719b229"} Oct 30 19:20:58.367749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:58.367746 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dzhlx" event={"ID":"2305471f-d5ac-43c7-b204-1650ae447ff9","Type":"ContainerStarted","Data":"31c793c7d73bd0d255f1886d79a0d3b1e0308c92b1ba609acbe92cb5a04d15ee"} Oct 30 19:20:58.369265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:58.369239 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-8ns8s" event={"ID":"70b0df95-c5a5-48b5-b670-8382137780ea","Type":"ContainerStarted","Data":"92e01c5a931ccc24c1011e38493ac185e98e320e0a3d3e523d854ba5168d5781"} Oct 30 19:20:58.369520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:58.369503 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:58.370642 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:58.370623 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56bc986487-ljfn2" Oct 30 19:20:58.999706 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:20:58.999673 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cf497f4fc-s2vpv"] Oct 30 19:21:03.755167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:03.755139 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:21:07.806417 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:07.806390 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f7zr5" Oct 30 19:21:08.640219 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.640188 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fcwl9"] Oct 30 19:21:08.681725 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.681700 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.706220 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.706201 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:21:08.706323 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.706290 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gqsl5\"" Oct 30 19:21:08.719494 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.719477 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:21:08.779827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.779807 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/029546f8-af87-4430-9de1-725b5e3389cf-sys\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.779915 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.779833 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/029546f8-af87-4430-9de1-725b5e3389cf-root\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.779915 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.779861 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.779915 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.779890 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.780044 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.780006 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/029546f8-af87-4430-9de1-725b5e3389cf-metrics-client-ca\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.780078 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.780053 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-textfile\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.780141 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.780076 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-wtmp\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.780141 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.780121 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-tls\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.780223 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.780211 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8j8\" (UniqueName: \"kubernetes.io/projected/029546f8-af87-4430-9de1-725b5e3389cf-kube-api-access-hr8j8\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.790627 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.790608 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:21:08.790712 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.790653 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.881499 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-wtmp\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.881676 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-wtmp\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.881733 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-tls\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.881769 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8j8\" (UniqueName: \"kubernetes.io/projected/029546f8-af87-4430-9de1-725b5e3389cf-kube-api-access-hr8j8\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.881796 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/029546f8-af87-4430-9de1-725b5e3389cf-sys\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.881819 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/029546f8-af87-4430-9de1-725b5e3389cf-root\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.881866 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.881898 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.881931 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/029546f8-af87-4430-9de1-725b5e3389cf-metrics-client-ca\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.881978 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-textfile\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.882188 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/029546f8-af87-4430-9de1-725b5e3389cf-root\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.882280 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/029546f8-af87-4430-9de1-725b5e3389cf-sys\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.882294 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-textfile\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.882773 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/029546f8-af87-4430-9de1-725b5e3389cf-metrics-client-ca\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.883130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.883026 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.884363 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.884340 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-tls\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.884918 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.884896 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/029546f8-af87-4430-9de1-725b5e3389cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:08.903300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:08.903251 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8j8\" (UniqueName: \"kubernetes.io/projected/029546f8-af87-4430-9de1-725b5e3389cf-kube-api-access-hr8j8\") pod \"node-exporter-fcwl9\" (UID: \"029546f8-af87-4430-9de1-725b5e3389cf\") " pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:09.005552 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:09.005534 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:21:09.034131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:09.034092 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcwl9" Oct 30 19:21:09.036788 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:09.036768 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zrrw2" Oct 30 19:21:09.078302 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:21:09.078281 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029546f8_af87_4430_9de1_725b5e3389cf.slice/crio-e012cd0ebea1d4fc0e8130a489961f33032874d7104347e4e353894693e410aa WatchSource:0}: Error finding container e012cd0ebea1d4fc0e8130a489961f33032874d7104347e4e353894693e410aa: Status 404 returned error can't find the container with id e012cd0ebea1d4fc0e8130a489961f33032874d7104347e4e353894693e410aa Oct 30 19:21:09.411213 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:09.411176 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcwl9" event={"ID":"029546f8-af87-4430-9de1-725b5e3389cf","Type":"ContainerStarted","Data":"e012cd0ebea1d4fc0e8130a489961f33032874d7104347e4e353894693e410aa"} Oct 30 19:21:11.421383 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:11.421350 4904 generic.go:356] "Generic (PLEG): container finished" podID="029546f8-af87-4430-9de1-725b5e3389cf" containerID="264aac5f626badd8ecc35128b570353049e317cffc05e02d3aaeeddca9ef7650" exitCode=0 Oct 30 19:21:11.421771 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:11.421440 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcwl9" event={"ID":"029546f8-af87-4430-9de1-725b5e3389cf","Type":"ContainerDied","Data":"264aac5f626badd8ecc35128b570353049e317cffc05e02d3aaeeddca9ef7650"} Oct 30 19:21:12.428394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:12.428356 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcwl9" event={"ID":"029546f8-af87-4430-9de1-725b5e3389cf","Type":"ContainerStarted","Data":"1726d78f9eb653e193821a6931dcbf922a00618a2c225cf494596b2f2ff89519"} Oct 30 19:21:12.428790 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:12.428414 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcwl9" event={"ID":"029546f8-af87-4430-9de1-725b5e3389cf","Type":"ContainerStarted","Data":"c616dcc372b2cfd1d90c2ba30b72d058bc2ce2a5ac84885ef8b1ad4013f08aac"} Oct 30 19:21:12.498771 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:12.498717 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fcwl9" podStartSLOduration=3.025607977 podStartE2EDuration="4.498704463s" totalImagesPullingTime="1.386497468s" totalInitContainerRuntime="86.599018ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:12.485071225 +0000 UTC m=+33.398675841" watchObservedRunningTime="2026-10-30 19:21:12.498704463 +0000 UTC m=+33.412309080" Oct 30 19:21:21.031982 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:21.031895 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d984dbd48-qljg6"] Oct 30 19:21:24.020374 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.020334 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" podUID="1697d46a-5b9e-429a-986f-76e472a662a5" containerName="registry" containerID="cri-o://57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87" gracePeriod=30 Oct 30 19:21:24.269653 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.269629 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:21:24.392704 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.392637 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-bound-sa-token\") pod \"1697d46a-5b9e-429a-986f-76e472a662a5\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " Oct 30 19:21:24.392704 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.392695 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-kube-api-access-lvkqc\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-kube-api-access-lvkqc\") pod \"1697d46a-5b9e-429a-986f-76e472a662a5\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " Oct 30 19:21:24.392885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.392715 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-registry-certificates\") pod \"1697d46a-5b9e-429a-986f-76e472a662a5\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " Oct 30 19:21:24.392885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.392750 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") pod \"1697d46a-5b9e-429a-986f-76e472a662a5\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " Oct 30 19:21:24.392885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.392771 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1697d46a-5b9e-429a-986f-76e472a662a5-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1697d46a-5b9e-429a-986f-76e472a662a5-ca-trust-extracted\") pod \"1697d46a-5b9e-429a-986f-76e472a662a5\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " Oct 30 19:21:24.392885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.392796 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-trusted-ca\") pod \"1697d46a-5b9e-429a-986f-76e472a662a5\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " Oct 30 19:21:24.392885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.392827 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-installation-pull-secrets\") pod \"1697d46a-5b9e-429a-986f-76e472a662a5\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " Oct 30 19:21:24.392885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.392853 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-image-registry-private-configuration\") pod \"1697d46a-5b9e-429a-986f-76e472a662a5\" (UID: \"1697d46a-5b9e-429a-986f-76e472a662a5\") " Oct 30 19:21:24.393315 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.393260 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-registry-certificates" pod "1697d46a-5b9e-429a-986f-76e472a662a5" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:24.393480 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.393440 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-trusted-ca" pod "1697d46a-5b9e-429a-986f-76e472a662a5" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:24.395605 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.395575 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-installation-pull-secrets" pod "1697d46a-5b9e-429a-986f-76e472a662a5" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:24.395706 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.395646 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-bound-sa-token" pod "1697d46a-5b9e-429a-986f-76e472a662a5" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:24.395706 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.395685 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls" pod "1697d46a-5b9e-429a-986f-76e472a662a5" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:24.395817 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.395758 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-kube-api-access-lvkqc" pod "1697d46a-5b9e-429a-986f-76e472a662a5" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5"). InnerVolumeSpecName "kube-api-access-lvkqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:24.395911 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.395892 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-image-registry-private-configuration" pod "1697d46a-5b9e-429a-986f-76e472a662a5" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:24.402773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.402714 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1697d46a-5b9e-429a-986f-76e472a662a5-ca-trust-extracted" pod "1697d46a-5b9e-429a-986f-76e472a662a5" (UID: "1697d46a-5b9e-429a-986f-76e472a662a5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:24.473032 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.473004 4904 generic.go:356] "Generic (PLEG): container finished" podID="1697d46a-5b9e-429a-986f-76e472a662a5" containerID="57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87" exitCode=0 Oct 30 19:21:24.473173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.473051 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" Oct 30 19:21:24.473173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.473052 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" event={"ID":"1697d46a-5b9e-429a-986f-76e472a662a5","Type":"ContainerDied","Data":"57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87"} Oct 30 19:21:24.473173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.473159 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf497f4fc-s2vpv" event={"ID":"1697d46a-5b9e-429a-986f-76e472a662a5","Type":"ContainerDied","Data":"98049724bbfc165f04e14d5c12416dbf85a0bdb56ea24070ffe2fe799ba2e1fa"} Oct 30 19:21:24.473276 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.473189 4904 scope.go:122] "RemoveContainer" containerID="57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87" Oct 30 19:21:24.481287 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.481270 4904 scope.go:122] "RemoveContainer" containerID="57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87" Oct 30 19:21:24.481546 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:21:24.481526 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87\": container with ID starting with 57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87 not found: ID does not exist" containerID="57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87" Oct 30 19:21:24.481626 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.481552 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87"} err="failed to get container status \"57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87\": rpc error: code = NotFound desc = could not find container \"57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87\": container with ID starting with 57453436a254d7e0208616673574d9463c39cc2e9bbe472cc2337045c48d5d87 not found: ID does not exist" Oct 30 19:21:24.493974 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.493948 4904 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-trusted-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:24.493974 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.493968 4904 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-installation-pull-secrets\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:24.493974 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.493978 4904 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1697d46a-5b9e-429a-986f-76e472a662a5-image-registry-private-configuration\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:24.494227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.493989 4904 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-bound-sa-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:24.494227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.493997 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvkqc\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-kube-api-access-lvkqc\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:24.494227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.494007 4904 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1697d46a-5b9e-429a-986f-76e472a662a5-registry-certificates\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:24.494227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.494015 4904 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1697d46a-5b9e-429a-986f-76e472a662a5-registry-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:24.494227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.494024 4904 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1697d46a-5b9e-429a-986f-76e472a662a5-ca-trust-extracted\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:24.507496 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:21:24.507470 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:21:24.527534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.527512 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cf497f4fc-s2vpv"] Oct 30 19:21:24.533802 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:24.533784 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cf497f4fc-s2vpv"] Oct 30 19:21:25.219559 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:25.219529 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1697d46a-5b9e-429a-986f-76e472a662a5" path="/var/lib/kubelet/pods/1697d46a-5b9e-429a-986f-76e472a662a5/volumes" Oct 30 19:21:45.553125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:45.553064 4904 generic.go:356] "Generic (PLEG): container finished" podID="dc69b2d3-eac5-43d1-86e4-36fa7cda7618" containerID="9d878bd241f9c9ea8b5fc82e3e8f12a8b3dd90fed2994013a35bb87cab4a7d4f" exitCode=0 Oct 30 19:21:45.553483 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:45.553141 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" event={"ID":"dc69b2d3-eac5-43d1-86e4-36fa7cda7618","Type":"ContainerDied","Data":"9d878bd241f9c9ea8b5fc82e3e8f12a8b3dd90fed2994013a35bb87cab4a7d4f"} Oct 30 19:21:45.553483 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:45.553437 4904 scope.go:122] "RemoveContainer" containerID="9d878bd241f9c9ea8b5fc82e3e8f12a8b3dd90fed2994013a35bb87cab4a7d4f" Oct 30 19:21:46.053528 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.053488 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" podUID="a4cf2efa-495e-452f-b0a6-b17c1d54038b" containerName="registry" containerID="cri-o://52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77" gracePeriod=30 Oct 30 19:21:46.287499 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.287474 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:21:46.344555 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.344497 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-certificates\") pod \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " Oct 30 19:21:46.344555 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.344550 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-kube-api-access-s5qrf\") pod \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " Oct 30 19:21:46.344709 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.344638 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cf2efa-495e-452f-b0a6-b17c1d54038b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4cf2efa-495e-452f-b0a6-b17c1d54038b-ca-trust-extracted\") pod \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " Oct 30 19:21:46.344709 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.344672 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-image-registry-private-configuration\") pod \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " Oct 30 19:21:46.344803 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.344758 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-trusted-ca\") pod \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " Oct 30 19:21:46.344858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.344797 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-installation-pull-secrets\") pod \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " Oct 30 19:21:46.344909 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.344832 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-bound-sa-token\") pod \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " Oct 30 19:21:46.344909 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.344905 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") pod \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\" (UID: \"a4cf2efa-495e-452f-b0a6-b17c1d54038b\") " Oct 30 19:21:46.345022 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.344864 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-certificates" pod "a4cf2efa-495e-452f-b0a6-b17c1d54038b" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:46.345396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.345191 4904 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-certificates\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:46.345618 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.345546 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-trusted-ca" pod "a4cf2efa-495e-452f-b0a6-b17c1d54038b" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:46.347014 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.346988 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-kube-api-access-s5qrf" pod "a4cf2efa-495e-452f-b0a6-b17c1d54038b" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b"). InnerVolumeSpecName "kube-api-access-s5qrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:46.347171 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.347153 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls" pod "a4cf2efa-495e-452f-b0a6-b17c1d54038b" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:46.347402 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.347384 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-bound-sa-token" pod "a4cf2efa-495e-452f-b0a6-b17c1d54038b" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:46.347722 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.347695 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-image-registry-private-configuration" pod "a4cf2efa-495e-452f-b0a6-b17c1d54038b" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:46.347796 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.347726 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-installation-pull-secrets" pod "a4cf2efa-495e-452f-b0a6-b17c1d54038b" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:46.351851 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.351831 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cf2efa-495e-452f-b0a6-b17c1d54038b-ca-trust-extracted" pod "a4cf2efa-495e-452f-b0a6-b17c1d54038b" (UID: "a4cf2efa-495e-452f-b0a6-b17c1d54038b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:46.446077 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.446055 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-kube-api-access-s5qrf\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:46.446077 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.446075 4904 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4cf2efa-495e-452f-b0a6-b17c1d54038b-ca-trust-extracted\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:46.446219 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.446085 4904 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-image-registry-private-configuration\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:46.446219 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.446095 4904 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4cf2efa-495e-452f-b0a6-b17c1d54038b-trusted-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:46.446219 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.446123 4904 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4cf2efa-495e-452f-b0a6-b17c1d54038b-installation-pull-secrets\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:46.446219 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.446132 4904 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-bound-sa-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:46.446219 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.446140 4904 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4cf2efa-495e-452f-b0a6-b17c1d54038b-registry-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:21:46.559115 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.559078 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qbhmk" event={"ID":"dc69b2d3-eac5-43d1-86e4-36fa7cda7618","Type":"ContainerStarted","Data":"67f6672d92c93df36ae6f9b33c463e63086b8937bb4a2354db550fa1bb6a4d70"} Oct 30 19:21:46.560289 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.560267 4904 generic.go:356] "Generic (PLEG): container finished" podID="a4cf2efa-495e-452f-b0a6-b17c1d54038b" containerID="52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77" exitCode=0 Oct 30 19:21:46.560378 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.560304 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" event={"ID":"a4cf2efa-495e-452f-b0a6-b17c1d54038b","Type":"ContainerDied","Data":"52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77"} Oct 30 19:21:46.560378 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.560316 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" Oct 30 19:21:46.560378 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.560332 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d984dbd48-qljg6" event={"ID":"a4cf2efa-495e-452f-b0a6-b17c1d54038b","Type":"ContainerDied","Data":"d295bf5871fc79f706a19478910dc30e18e9175e2cce49f9350ee37deb999357"} Oct 30 19:21:46.560378 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.560359 4904 scope.go:122] "RemoveContainer" containerID="52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77" Oct 30 19:21:46.571925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.570908 4904 scope.go:122] "RemoveContainer" containerID="52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77" Oct 30 19:21:46.571925 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:21:46.571317 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77\": container with ID starting with 52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77 not found: ID does not exist" containerID="52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77" Oct 30 19:21:46.571925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.571348 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77"} err="failed to get container status \"52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77\": rpc error: code = NotFound desc = could not find container \"52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77\": container with ID starting with 52d290dc37690eaa0114bef27d539cf8e62efb760630d712f46d91e008dbab77 not found: ID does not exist" Oct 30 19:21:46.637949 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.637894 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d984dbd48-qljg6"] Oct 30 19:21:46.651283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:46.651262 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d984dbd48-qljg6"] Oct 30 19:21:47.220036 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:21:47.220006 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4cf2efa-495e-452f-b0a6-b17c1d54038b" path="/var/lib/kubelet/pods/a4cf2efa-495e-452f-b0a6-b17c1d54038b/volumes" Oct 30 19:21:54.508223 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:21:54.508188 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:22:00.615813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:22:00.615784 4904 generic.go:356] "Generic (PLEG): container finished" podID="2e8615b0-abab-426b-a0ee-893109807207" containerID="88ef7abcedff83cde2ba656b6d6e3482c333e09db5b0c7a1b3aed57f0feda0fa" exitCode=0 Oct 30 19:22:00.616194 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:22:00.615861 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-58fgj" event={"ID":"2e8615b0-abab-426b-a0ee-893109807207","Type":"ContainerDied","Data":"88ef7abcedff83cde2ba656b6d6e3482c333e09db5b0c7a1b3aed57f0feda0fa"} Oct 30 19:22:00.616194 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:22:00.616160 4904 scope.go:122] "RemoveContainer" containerID="88ef7abcedff83cde2ba656b6d6e3482c333e09db5b0c7a1b3aed57f0feda0fa" Oct 30 19:22:01.620653 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:22:01.620625 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-58fgj" event={"ID":"2e8615b0-abab-426b-a0ee-893109807207","Type":"ContainerStarted","Data":"043e6224513c12bede2c1aaf70bbbe117bff2aad35c7ad6ac21e7f48ce45c2fb"} Oct 30 19:22:15.671605 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:22:15.671571 4904 generic.go:356] "Generic (PLEG): container finished" podID="f8e59f23-4988-41f7-b462-912ac6c7c125" containerID="89332f550d212284a9adc7949f42f5deb7ce0bbb5775f8e5db5b3c9534f4355a" exitCode=0 Oct 30 19:22:15.671955 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:22:15.671645 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" event={"ID":"f8e59f23-4988-41f7-b462-912ac6c7c125","Type":"ContainerDied","Data":"89332f550d212284a9adc7949f42f5deb7ce0bbb5775f8e5db5b3c9534f4355a"} Oct 30 19:22:15.671955 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:22:15.671945 4904 scope.go:122] "RemoveContainer" containerID="89332f550d212284a9adc7949f42f5deb7ce0bbb5775f8e5db5b3c9534f4355a" Oct 30 19:22:16.677132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:22:16.677081 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-scc42" event={"ID":"f8e59f23-4988-41f7-b462-912ac6c7c125","Type":"ContainerStarted","Data":"672ae39d0802936dcc865eb76b790e8f7657e3748f4d3b4aa6c85882cb057223"} Oct 30 19:22:24.508552 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:22:24.508519 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:22:54.508585 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:22:54.508505 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:23:24.508202 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:23:24.508167 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:23:54.508285 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:23:54.508251 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:24:24.507531 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:24:24.507457 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:24:32.967086 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:32.967057 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5"] Oct 30 19:24:32.967638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:32.967450 4904 cpu_manager.go:414] "containerMap: removing container" podUID="a4cf2efa-495e-452f-b0a6-b17c1d54038b" containerName="registry" Oct 30 19:24:32.967638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:32.967468 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4cf2efa-495e-452f-b0a6-b17c1d54038b" containerName="registry" Oct 30 19:24:32.967638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:32.967493 4904 cpu_manager.go:414] "containerMap: removing container" podUID="1697d46a-5b9e-429a-986f-76e472a662a5" containerName="registry" Oct 30 19:24:32.967638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:32.967500 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1697d46a-5b9e-429a-986f-76e472a662a5" containerName="registry" Oct 30 19:24:32.967638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:32.967576 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="1697d46a-5b9e-429a-986f-76e472a662a5" containerName="registry" Oct 30 19:24:32.967638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:32.967590 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4cf2efa-495e-452f-b0a6-b17c1d54038b" containerName="registry" Oct 30 19:24:32.970696 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:32.970678 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.000636 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.000605 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-dn88r\"" Oct 30 19:24:33.000752 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.000710 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:24:33.000814 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.000767 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:24:33.001586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.001567 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5"] Oct 30 19:24:33.010146 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.010118 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r98x\" (UniqueName: \"kubernetes.io/projected/93086c4a-df9c-43ec-931d-05f425b67a01-kube-api-access-8r98x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.010239 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.010205 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.010300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.010285 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.110938 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.110918 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.111018 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.110953 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r98x\" (UniqueName: \"kubernetes.io/projected/93086c4a-df9c-43ec-931d-05f425b67a01-kube-api-access-8r98x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.111018 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.110975 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.111253 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.111239 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.111314 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.111295 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.132524 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.132504 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r98x\" (UniqueName: \"kubernetes.io/projected/93086c4a-df9c-43ec-931d-05f425b67a01-kube-api-access-8r98x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.287835 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.287785 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:24:33.585218 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.585152 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5"] Oct 30 19:24:33.588250 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:24:33.588227 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93086c4a_df9c_43ec_931d_05f425b67a01.slice/crio-e1143985368fff477cacb45d6c743f40bceccce705b7a3c0cf414a15ca118f06 WatchSource:0}: Error finding container e1143985368fff477cacb45d6c743f40bceccce705b7a3c0cf414a15ca118f06: Status 404 returned error can't find the container with id e1143985368fff477cacb45d6c743f40bceccce705b7a3c0cf414a15ca118f06 Oct 30 19:24:33.636511 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.636485 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q"] Oct 30 19:24:33.640869 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.640854 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:33.660247 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.660225 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q"] Oct 30 19:24:33.700194 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.700178 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-xphz4\"" Oct 30 19:24:33.716034 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.716014 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:33.716092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.716040 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:33.716092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.716070 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhxtx\" (UniqueName: \"kubernetes.io/projected/4ce48b21-3d85-4209-bb18-47637478001f-kube-api-access-bhxtx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:33.817345 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.817323 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:33.817429 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.817352 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:33.817429 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.817379 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxtx\" (UniqueName: \"kubernetes.io/projected/4ce48b21-3d85-4209-bb18-47637478001f-kube-api-access-bhxtx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:33.817677 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.817660 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:33.817714 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.817692 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:33.838220 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.838170 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxtx\" (UniqueName: \"kubernetes.io/projected/4ce48b21-3d85-4209-bb18-47637478001f-kube-api-access-bhxtx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:33.955663 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:33.955643 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:24:34.174427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.174395 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" event={"ID":"93086c4a-df9c-43ec-931d-05f425b67a01","Type":"ContainerStarted","Data":"e1143985368fff477cacb45d6c743f40bceccce705b7a3c0cf414a15ca118f06"} Oct 30 19:24:34.224326 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.224301 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm"] Oct 30 19:24:34.227964 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.227946 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.247417 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.247392 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm"] Oct 30 19:24:34.258708 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.258683 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-ps27s\"" Oct 30 19:24:34.275225 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.275201 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q"] Oct 30 19:24:34.279226 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:24:34.279202 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce48b21_3d85_4209_bb18_47637478001f.slice/crio-12d1cb0a0657d756c1815c44cf99f650531ab87e1268b9ed8fbd489b763a7155 WatchSource:0}: Error finding container 12d1cb0a0657d756c1815c44cf99f650531ab87e1268b9ed8fbd489b763a7155: Status 404 returned error can't find the container with id 12d1cb0a0657d756c1815c44cf99f650531ab87e1268b9ed8fbd489b763a7155 Oct 30 19:24:34.319175 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.319149 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499"] Oct 30 19:24:34.322536 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.322509 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.322676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.322659 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g5nt\" (UniqueName: \"kubernetes.io/projected/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-kube-api-access-8g5nt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.322761 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.322691 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.322761 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.322722 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.341910 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.341887 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499"] Oct 30 19:24:34.346489 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.346469 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-gz5z7\"" Oct 30 19:24:34.423889 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.423862 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g5nt\" (UniqueName: \"kubernetes.io/projected/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-kube-api-access-8g5nt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.424021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.423904 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.424021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.423935 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.424021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.423971 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.424021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.424015 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp5t2\" (UniqueName: \"kubernetes.io/projected/91687dd6-1138-46ad-92e6-eff952029d7c-kube-api-access-cp5t2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.424342 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.424076 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.424877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.424840 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.425049 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.425033 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.426506 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.426484 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n"] Oct 30 19:24:34.429960 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.429944 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.448597 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.448573 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g5nt\" (UniqueName: \"kubernetes.io/projected/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-kube-api-access-8g5nt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.448810 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.448792 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-rltxw\"" Oct 30 19:24:34.448868 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.448803 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n"] Oct 30 19:24:34.519264 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.519236 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf"] Oct 30 19:24:34.522935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.522914 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.525430 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.525404 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp5t2\" (UniqueName: \"kubernetes.io/projected/91687dd6-1138-46ad-92e6-eff952029d7c-kube-api-access-cp5t2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.525556 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.525459 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.525556 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.525493 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.525556 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.525522 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.525722 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.525570 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwqs\" (UniqueName: \"kubernetes.io/projected/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-kube-api-access-8jwqs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.525722 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.525602 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.525977 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.525956 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.526298 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.526275 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.528450 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.528423 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5"] Oct 30 19:24:34.531816 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.531796 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.540368 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.540347 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf"] Oct 30 19:24:34.547151 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.546645 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:24:34.547151 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.546661 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp5t2\" (UniqueName: \"kubernetes.io/projected/91687dd6-1138-46ad-92e6-eff952029d7c-kube-api-access-cp5t2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.547151 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.546961 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-tsd4c\"" Oct 30 19:24:34.548795 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.548740 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5"] Oct 30 19:24:34.626420 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.626352 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.626420 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.626397 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.626599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.626432 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.626599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.626472 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.626599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.626506 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q57b4\" (UniqueName: \"kubernetes.io/projected/a016102a-5616-43c4-b5d5-97a01cf4774c-kube-api-access-q57b4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.626599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.626535 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwqs\" (UniqueName: \"kubernetes.io/projected/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-kube-api-access-8jwqs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.626599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.626591 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.626852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.626655 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnmh\" (UniqueName: \"kubernetes.io/projected/5e27b37b-33f6-4e07-b278-942cf362fe2b-kube-api-access-qxnmh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.626852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.626689 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.627093 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.627060 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.628009 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.627927 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.640961 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.640940 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:24:34.647525 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.647505 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-cdmkt\"" Oct 30 19:24:34.652865 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.652788 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwqs\" (UniqueName: \"kubernetes.io/projected/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-kube-api-access-8jwqs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.728202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.727643 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.728202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.727713 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnmh\" (UniqueName: \"kubernetes.io/projected/5e27b37b-33f6-4e07-b278-942cf362fe2b-kube-api-access-qxnmh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.728202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.727754 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.728202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.727778 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.728202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.727809 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.728202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.727839 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q57b4\" (UniqueName: \"kubernetes.io/projected/a016102a-5616-43c4-b5d5-97a01cf4774c-kube-api-access-q57b4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.728202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.728119 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.728606 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.728442 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.728606 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.728498 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.728695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.728671 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.746363 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.746184 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:24:34.750460 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.750423 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnmh\" (UniqueName: \"kubernetes.io/projected/5e27b37b-33f6-4e07-b278-942cf362fe2b-kube-api-access-qxnmh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.752002 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.751983 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q57b4\" (UniqueName: \"kubernetes.io/projected/a016102a-5616-43c4-b5d5-97a01cf4774c-kube-api-access-q57b4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.842825 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.842796 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:24:34.850495 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.850472 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:24:34.857460 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.856010 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm"] Oct 30 19:24:34.862950 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:24:34.862924 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e17c58c_b2b2_48a9_b28b_0afad316a0d8.slice/crio-53c5a3dea3c80771d4603398fc3e889e56b220b77d59bf5f05108ed53a427a7d WatchSource:0}: Error finding container 53c5a3dea3c80771d4603398fc3e889e56b220b77d59bf5f05108ed53a427a7d: Status 404 returned error can't find the container with id 53c5a3dea3c80771d4603398fc3e889e56b220b77d59bf5f05108ed53a427a7d Oct 30 19:24:34.965577 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:34.965487 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499"] Oct 30 19:24:34.966544 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:24:34.966514 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91687dd6_1138_46ad_92e6_eff952029d7c.slice/crio-c637137adcaf8ec3e4277e04c6a566deeafdef819b9ebca1c30fdd6ad0a9368c WatchSource:0}: Error finding container c637137adcaf8ec3e4277e04c6a566deeafdef819b9ebca1c30fdd6ad0a9368c: Status 404 returned error can't find the container with id c637137adcaf8ec3e4277e04c6a566deeafdef819b9ebca1c30fdd6ad0a9368c Oct 30 19:24:35.061550 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:35.061520 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n"] Oct 30 19:24:35.065322 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:24:35.065286 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fe3bf8_5c19_4bac_ad0c_1d31d2dc1753.slice/crio-1a90fdd0fa37ac463aa045cbd3d5877aec95c102682682c55c81e059a98d57c7 WatchSource:0}: Error finding container 1a90fdd0fa37ac463aa045cbd3d5877aec95c102682682c55c81e059a98d57c7: Status 404 returned error can't find the container with id 1a90fdd0fa37ac463aa045cbd3d5877aec95c102682682c55c81e059a98d57c7 Oct 30 19:24:35.177047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:35.177020 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf"] Oct 30 19:24:35.180709 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:24:35.180682 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e27b37b_33f6_4e07_b278_942cf362fe2b.slice/crio-36313f808d4a78bd7c5f4b3612ad610afff1be4060d144376349f194ed910f27 WatchSource:0}: Error finding container 36313f808d4a78bd7c5f4b3612ad610afff1be4060d144376349f194ed910f27: Status 404 returned error can't find the container with id 36313f808d4a78bd7c5f4b3612ad610afff1be4060d144376349f194ed910f27 Oct 30 19:24:35.181074 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:35.181045 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" event={"ID":"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753","Type":"ContainerStarted","Data":"1a90fdd0fa37ac463aa045cbd3d5877aec95c102682682c55c81e059a98d57c7"} Oct 30 19:24:35.183871 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:35.183801 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" event={"ID":"4ce48b21-3d85-4209-bb18-47637478001f","Type":"ContainerStarted","Data":"12d1cb0a0657d756c1815c44cf99f650531ab87e1268b9ed8fbd489b763a7155"} Oct 30 19:24:35.185578 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:35.185514 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" event={"ID":"91687dd6-1138-46ad-92e6-eff952029d7c","Type":"ContainerStarted","Data":"c637137adcaf8ec3e4277e04c6a566deeafdef819b9ebca1c30fdd6ad0a9368c"} Oct 30 19:24:35.186832 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:35.186807 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" event={"ID":"1e17c58c-b2b2-48a9-b28b-0afad316a0d8","Type":"ContainerStarted","Data":"53c5a3dea3c80771d4603398fc3e889e56b220b77d59bf5f05108ed53a427a7d"} Oct 30 19:24:35.190854 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:35.190828 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5"] Oct 30 19:24:35.193915 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:24:35.193895 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda016102a_5616_43c4_b5d5_97a01cf4774c.slice/crio-128483f0b129cffb55bd3516242efec6b50ba2260269cf53145394b95020906b WatchSource:0}: Error finding container 128483f0b129cffb55bd3516242efec6b50ba2260269cf53145394b95020906b: Status 404 returned error can't find the container with id 128483f0b129cffb55bd3516242efec6b50ba2260269cf53145394b95020906b Oct 30 19:24:36.194721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:36.194649 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" event={"ID":"a016102a-5616-43c4-b5d5-97a01cf4774c","Type":"ContainerStarted","Data":"128483f0b129cffb55bd3516242efec6b50ba2260269cf53145394b95020906b"} Oct 30 19:24:36.196454 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:36.196365 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" event={"ID":"5e27b37b-33f6-4e07-b278-942cf362fe2b","Type":"ContainerStarted","Data":"36313f808d4a78bd7c5f4b3612ad610afff1be4060d144376349f194ed910f27"} Oct 30 19:24:37.442929 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.441816 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj"] Oct 30 19:24:37.445728 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.445648 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.466944 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.466880 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj"] Oct 30 19:24:37.467866 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.467814 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:24:37.488610 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.488574 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-q4g2l\"" Oct 30 19:24:37.554045 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.553494 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.554045 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.553536 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdhrv\" (UniqueName: \"kubernetes.io/projected/0c8a2a4b-6240-40fa-a873-d2300e317a2b-kube-api-access-sdhrv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.554045 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.553581 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.554045 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.553612 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.554045 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.553637 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.554045 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.553674 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.554045 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.553702 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.554045 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.553732 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.554045 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.553781 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.570399 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.568671 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk"] Oct 30 19:24:37.573288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.572593 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.591847 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.591814 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk"] Oct 30 19:24:37.593359 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.593334 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-7zbjk\"" Oct 30 19:24:37.654367 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654335 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.654500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654380 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.654500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654424 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.654500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654451 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.654500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654481 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.654721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654532 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.654721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654585 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.654721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654614 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.654721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654640 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxj7\" (UniqueName: \"kubernetes.io/projected/c06db63b-4ed7-499f-92e8-9f8ebd75598b-kube-api-access-ttxj7\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.654721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654677 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.654721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654711 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.655021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654738 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.655021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654787 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.655021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654812 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdhrv\" (UniqueName: \"kubernetes.io/projected/0c8a2a4b-6240-40fa-a873-d2300e317a2b-kube-api-access-sdhrv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.655021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654847 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.655021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654886 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.655021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654937 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.655021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.654961 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.655465 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.655435 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.656235 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.656213 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.656490 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.656468 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.657794 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.657770 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.658063 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.658025 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.663274 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.663207 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.665709 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.665677 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.688816 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.688790 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c8a2a4b-6240-40fa-a873-d2300e317a2b-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.704691 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.704633 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdhrv\" (UniqueName: \"kubernetes.io/projected/0c8a2a4b-6240-40fa-a873-d2300e317a2b-kube-api-access-sdhrv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ggdsj\" (UID: \"0c8a2a4b-6240-40fa-a873-d2300e317a2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.755995 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.756048 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.756073 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.756169 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.756204 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.756236 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxj7\" (UniqueName: \"kubernetes.io/projected/c06db63b-4ed7-499f-92e8-9f8ebd75598b-kube-api-access-ttxj7\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.756273 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.756309 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.756336 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.757119 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.757501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.757385 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.758453 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.758428 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.758711 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.758692 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.758935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.758916 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.766206 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.766184 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:37.773004 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.772979 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.773223 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.773201 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.779952 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.779931 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c06db63b-4ed7-499f-92e8-9f8ebd75598b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.783085 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.783059 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxj7\" (UniqueName: \"kubernetes.io/projected/c06db63b-4ed7-499f-92e8-9f8ebd75598b-kube-api-access-ttxj7\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk\" (UID: \"c06db63b-4ed7-499f-92e8-9f8ebd75598b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:37.890306 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:37.890265 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:38.071408 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.071375 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6"] Oct 30 19:24:38.076965 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.076566 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.118145 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.116829 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6"] Oct 30 19:24:38.139646 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.139618 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj"] Oct 30 19:24:38.163389 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.163359 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.163511 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.163418 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x524q\" (UniqueName: \"kubernetes.io/projected/7aa41309-8972-458c-858e-678ef79c375e-kube-api-access-x524q\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.163511 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.163489 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.218698 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.218609 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" event={"ID":"0c8a2a4b-6240-40fa-a873-d2300e317a2b","Type":"ContainerStarted","Data":"ceed3b572e86dfd54e2fc7fa57eea512c5c68c3b97dbb7354687a1c636947262"} Oct 30 19:24:38.221294 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.221271 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk"] Oct 30 19:24:38.230178 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:24:38.230138 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06db63b_4ed7_499f_92e8_9f8ebd75598b.slice/crio-9bf4ae1bbe31543d1f8bcc4741a125b089dcaebd91a6946c86b5733cfdbeb7c7 WatchSource:0}: Error finding container 9bf4ae1bbe31543d1f8bcc4741a125b089dcaebd91a6946c86b5733cfdbeb7c7: Status 404 returned error can't find the container with id 9bf4ae1bbe31543d1f8bcc4741a125b089dcaebd91a6946c86b5733cfdbeb7c7 Oct 30 19:24:38.264540 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.264504 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.264696 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.264590 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.264696 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.264649 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x524q\" (UniqueName: \"kubernetes.io/projected/7aa41309-8972-458c-858e-678ef79c375e-kube-api-access-x524q\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.265290 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.265230 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.265496 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.265476 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.297131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.296974 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x524q\" (UniqueName: \"kubernetes.io/projected/7aa41309-8972-458c-858e-678ef79c375e-kube-api-access-x524q\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.397319 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.397283 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:24:38.731839 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:38.731581 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6"] Oct 30 19:24:38.735903 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:24:38.735723 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa41309_8972_458c_858e_678ef79c375e.slice/crio-3e6ecca7d081f9327b2760115685e0ab50a14a36ae778373e92e85827ea7ca69 WatchSource:0}: Error finding container 3e6ecca7d081f9327b2760115685e0ab50a14a36ae778373e92e85827ea7ca69: Status 404 returned error can't find the container with id 3e6ecca7d081f9327b2760115685e0ab50a14a36ae778373e92e85827ea7ca69 Oct 30 19:24:39.247358 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.247269 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" event={"ID":"c06db63b-4ed7-499f-92e8-9f8ebd75598b","Type":"ContainerStarted","Data":"9bf4ae1bbe31543d1f8bcc4741a125b089dcaebd91a6946c86b5733cfdbeb7c7"} Oct 30 19:24:39.263317 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.263277 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" event={"ID":"7aa41309-8972-458c-858e-678ef79c375e","Type":"ContainerStarted","Data":"3e6ecca7d081f9327b2760115685e0ab50a14a36ae778373e92e85827ea7ca69"} Oct 30 19:24:39.653814 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.653741 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr"] Oct 30 19:24:39.660936 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.660902 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:39.682997 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.679016 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr"] Oct 30 19:24:39.692248 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.690457 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd"] Oct 30 19:24:39.709215 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.708002 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd"] Oct 30 19:24:39.709215 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.708124 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l"] Oct 30 19:24:39.709215 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.708689 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:39.720287 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.718913 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:39.727877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.727818 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l"] Oct 30 19:24:39.746775 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.746738 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh"] Oct 30 19:24:39.757937 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.756427 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:39.763369 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.763215 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh"] Oct 30 19:24:39.783873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.782336 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:39.783873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.782413 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:39.783873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.782471 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nxc\" (UniqueName: \"kubernetes.io/projected/550c652c-4e47-487a-ae4e-1324b8d44cf4-kube-api-access-l7nxc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:39.783873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.782565 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:39.783873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.782610 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:39.783873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.782644 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:39.783873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.782679 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sj8h\" (UniqueName: \"kubernetes.io/projected/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-kube-api-access-8sj8h\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:39.783873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.782742 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:39.783873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.782779 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lspzv\" (UniqueName: \"kubernetes.io/projected/936a4b81-9973-4994-a6a9-6e7e132c35cb-kube-api-access-lspzv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.883918 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.883964 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nxc\" (UniqueName: \"kubernetes.io/projected/550c652c-4e47-487a-ae4e-1324b8d44cf4-kube-api-access-l7nxc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884010 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884026 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884043 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884061 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sj8h\" (UniqueName: \"kubernetes.io/projected/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-kube-api-access-8sj8h\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884078 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884124 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884146 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspzv\" (UniqueName: \"kubernetes.io/projected/936a4b81-9973-4994-a6a9-6e7e132c35cb-kube-api-access-lspzv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884181 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884234 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884260 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh2mk\" (UniqueName: \"kubernetes.io/projected/edcb47a2-a713-4645-9c7b-3e8441a28055-kube-api-access-qh2mk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.884918 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.885253 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.885474 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.886020 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.886522 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:39.886801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.886755 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:39.914385 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.914260 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspzv\" (UniqueName: \"kubernetes.io/projected/936a4b81-9973-4994-a6a9-6e7e132c35cb-kube-api-access-lspzv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:39.920810 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.920751 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sj8h\" (UniqueName: \"kubernetes.io/projected/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-kube-api-access-8sj8h\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:39.926794 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.926766 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nxc\" (UniqueName: \"kubernetes.io/projected/550c652c-4e47-487a-ae4e-1324b8d44cf4-kube-api-access-l7nxc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:39.985625 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.985593 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:39.985782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.985665 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh2mk\" (UniqueName: \"kubernetes.io/projected/edcb47a2-a713-4645-9c7b-3e8441a28055-kube-api-access-qh2mk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:39.985782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.985684 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:39.986003 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.985984 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:39.986237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.986214 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:39.993504 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:39.993482 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:24:40.010269 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:40.010216 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh2mk\" (UniqueName: \"kubernetes.io/projected/edcb47a2-a713-4645-9c7b-3e8441a28055-kube-api-access-qh2mk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:40.065613 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:40.065397 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:24:40.081962 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:40.081763 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:24:40.094134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:40.094073 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:24:40.400632 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:40.400520 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd"] Oct 30 19:24:40.417830 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:40.416382 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr"] Oct 30 19:24:40.456453 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:40.456409 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh"] Oct 30 19:24:40.488881 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:40.488700 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l"] Oct 30 19:24:40.493113 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:24:40.493051 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936a4b81_9973_4994_a6a9_6e7e132c35cb.slice/crio-655ef768a81dc18d0c497ed73b9c9340a694e8f77d86243b9465ba996cdd55a0 WatchSource:0}: Error finding container 655ef768a81dc18d0c497ed73b9c9340a694e8f77d86243b9465ba996cdd55a0: Status 404 returned error can't find the container with id 655ef768a81dc18d0c497ed73b9c9340a694e8f77d86243b9465ba996cdd55a0 Oct 30 19:24:41.319660 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:41.319556 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" event={"ID":"550c652c-4e47-487a-ae4e-1324b8d44cf4","Type":"ContainerStarted","Data":"005f17fa92031d59918432c1b5898eb2fbc6fa443c0e6c958e9a04b4469f3647"} Oct 30 19:24:41.332639 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:41.332543 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" event={"ID":"af15f75c-a27c-4582-b18b-7b6c9eeea6aa","Type":"ContainerStarted","Data":"6d861e6b4d1f09c5471ce9f91c590262b5ad78ee0a7545ad73b20d86fbdf66aa"} Oct 30 19:24:41.365174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:41.365138 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" event={"ID":"936a4b81-9973-4994-a6a9-6e7e132c35cb","Type":"ContainerStarted","Data":"655ef768a81dc18d0c497ed73b9c9340a694e8f77d86243b9465ba996cdd55a0"} Oct 30 19:24:41.377843 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:41.377811 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" event={"ID":"edcb47a2-a713-4645-9c7b-3e8441a28055","Type":"ContainerStarted","Data":"90ce99e5ed92eca3650af918fc7f2c10c3420664559f47f3a1b500b594d70461"} Oct 30 19:24:50.459142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:50.458945 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:24:50.459142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:50.459039 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:24:50.460569 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:50.459091 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:24:51.057721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:51.057671 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:24:51.057842 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:51.057777 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:24:51.057842 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:51.057822 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:24:51.566196 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:51.565606 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" event={"ID":"0c8a2a4b-6240-40fa-a873-d2300e317a2b","Type":"ContainerStarted","Data":"96668163d9dee1f37e1451e6e6a8c0b37f140edf9445dc51b304cc447019d47e"} Oct 30 19:24:51.572096 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:51.571438 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" event={"ID":"c06db63b-4ed7-499f-92e8-9f8ebd75598b","Type":"ContainerStarted","Data":"66b464a288777e5b4a80bbadcb1bb671da9be0ea3207be0480ce1b70a0fe7af4"} Oct 30 19:24:51.629807 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:51.629588 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" podStartSLOduration=2.333662352 podStartE2EDuration="14.629408616s" totalImagesPullingTime="12.295746264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:51.627366286 +0000 UTC m=+252.540970903" watchObservedRunningTime="2026-10-30 19:24:51.629408616 +0000 UTC m=+252.543013234" Oct 30 19:24:51.675345 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:51.675283 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" podStartSLOduration=1.85180716 podStartE2EDuration="14.675269236s" totalImagesPullingTime="12.823462076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:51.673495837 +0000 UTC m=+252.587100454" watchObservedRunningTime="2026-10-30 19:24:51.675269236 +0000 UTC m=+252.588873855" Oct 30 19:24:51.768163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:51.766948 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:51.773724 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:51.773487 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:51.890843 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:51.890712 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:52.578782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:52.578748 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:52.579844 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:52.579822 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:24:52.890801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:52.890728 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:52.897193 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:52.897167 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:53.587270 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:53.587242 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk" Oct 30 19:24:54.591420 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:24:54.591348 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:24:55.610481 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:55.610448 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" event={"ID":"550c652c-4e47-487a-ae4e-1324b8d44cf4","Type":"ContainerStarted","Data":"b46077d700a43fcd165f9896ea25d14996596d36f13d35cba45e14ee2793f67b"} Oct 30 19:24:55.613532 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:55.613504 4904 generic.go:356] "Generic (PLEG): container finished" podID="93086c4a-df9c-43ec-931d-05f425b67a01" containerID="256615f78d85b480a8fad5c041800e392cd76be24f1f02c3271a1a8e7126d24e" exitCode=0 Oct 30 19:24:55.613646 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:55.613604 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" event={"ID":"93086c4a-df9c-43ec-931d-05f425b67a01","Type":"ContainerDied","Data":"256615f78d85b480a8fad5c041800e392cd76be24f1f02c3271a1a8e7126d24e"} Oct 30 19:24:56.631214 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:56.631179 4904 generic.go:356] "Generic (PLEG): container finished" podID="550c652c-4e47-487a-ae4e-1324b8d44cf4" containerID="b46077d700a43fcd165f9896ea25d14996596d36f13d35cba45e14ee2793f67b" exitCode=0 Oct 30 19:24:56.631623 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:56.631262 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" event={"ID":"550c652c-4e47-487a-ae4e-1324b8d44cf4","Type":"ContainerDied","Data":"b46077d700a43fcd165f9896ea25d14996596d36f13d35cba45e14ee2793f67b"} Oct 30 19:24:56.636644 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:56.636588 4904 generic.go:356] "Generic (PLEG): container finished" podID="af15f75c-a27c-4582-b18b-7b6c9eeea6aa" containerID="37caa230e2cf1b6b6a27e89f14d3a125cd790f7275de4ff5ea4dca8763feeb33" exitCode=0 Oct 30 19:24:56.636752 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:56.636679 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" event={"ID":"af15f75c-a27c-4582-b18b-7b6c9eeea6aa","Type":"ContainerDied","Data":"37caa230e2cf1b6b6a27e89f14d3a125cd790f7275de4ff5ea4dca8763feeb33"} Oct 30 19:24:56.641391 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:56.641303 4904 generic.go:356] "Generic (PLEG): container finished" podID="936a4b81-9973-4994-a6a9-6e7e132c35cb" containerID="7ec57ae43b76a8563d250b161c4d65579a1c90e769e498c5cebac53e933cc31f" exitCode=0 Oct 30 19:24:56.641507 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:56.641409 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" event={"ID":"936a4b81-9973-4994-a6a9-6e7e132c35cb","Type":"ContainerDied","Data":"7ec57ae43b76a8563d250b161c4d65579a1c90e769e498c5cebac53e933cc31f"} Oct 30 19:24:58.349050 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:57.651564 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" event={"ID":"1e17c58c-b2b2-48a9-b28b-0afad316a0d8","Type":"ContainerStarted","Data":"258d15834d54538d8b6d9997085017a6ebf7e6f05c691c2b581f40a9439b23dc"} Oct 30 19:24:58.349050 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:57.655527 4904 generic.go:356] "Generic (PLEG): container finished" podID="4ce48b21-3d85-4209-bb18-47637478001f" containerID="c904d9e9b7b7e2ea33cb7452f2e506d453f7a60f515aab5a2e14942eb10042f5" exitCode=0 Oct 30 19:24:58.349050 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:57.655638 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" event={"ID":"4ce48b21-3d85-4209-bb18-47637478001f","Type":"ContainerDied","Data":"c904d9e9b7b7e2ea33cb7452f2e506d453f7a60f515aab5a2e14942eb10042f5"} Oct 30 19:24:58.663267 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:58.662635 4904 generic.go:356] "Generic (PLEG): container finished" podID="a016102a-5616-43c4-b5d5-97a01cf4774c" containerID="a29fa44472a491de381893b2adad3e2ab2976663f56090ce3774005fecf6e637" exitCode=0 Oct 30 19:24:58.663267 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:58.662713 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" event={"ID":"a016102a-5616-43c4-b5d5-97a01cf4774c","Type":"ContainerDied","Data":"a29fa44472a491de381893b2adad3e2ab2976663f56090ce3774005fecf6e637"} Oct 30 19:24:58.666018 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:58.665439 4904 generic.go:356] "Generic (PLEG): container finished" podID="5e27b37b-33f6-4e07-b278-942cf362fe2b" containerID="71a2c51b0fdee0e30684a56f254c5cf752671bc6afa671d8caa0df17a3ebff08" exitCode=0 Oct 30 19:24:58.666018 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:58.665538 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" event={"ID":"5e27b37b-33f6-4e07-b278-942cf362fe2b","Type":"ContainerDied","Data":"71a2c51b0fdee0e30684a56f254c5cf752671bc6afa671d8caa0df17a3ebff08"} Oct 30 19:24:58.669988 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:58.669876 4904 generic.go:356] "Generic (PLEG): container finished" podID="91687dd6-1138-46ad-92e6-eff952029d7c" containerID="498b3bfde9af10956baf2d857522696b17d3a6eeb47f2ca9c32f004116fe9805" exitCode=0 Oct 30 19:24:58.669988 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:58.669964 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" event={"ID":"91687dd6-1138-46ad-92e6-eff952029d7c","Type":"ContainerDied","Data":"498b3bfde9af10956baf2d857522696b17d3a6eeb47f2ca9c32f004116fe9805"} Oct 30 19:24:58.673074 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:58.672578 4904 generic.go:356] "Generic (PLEG): container finished" podID="1e17c58c-b2b2-48a9-b28b-0afad316a0d8" containerID="258d15834d54538d8b6d9997085017a6ebf7e6f05c691c2b581f40a9439b23dc" exitCode=0 Oct 30 19:24:58.673074 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:58.672617 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" event={"ID":"1e17c58c-b2b2-48a9-b28b-0afad316a0d8","Type":"ContainerDied","Data":"258d15834d54538d8b6d9997085017a6ebf7e6f05c691c2b581f40a9439b23dc"} Oct 30 19:24:59.678569 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:59.678535 4904 generic.go:356] "Generic (PLEG): container finished" podID="42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" containerID="3dad3047e8818474ff1b0e45fea437a34eb638f96303df853dd1a4ce36ee969b" exitCode=0 Oct 30 19:24:59.678918 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:59.678620 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" event={"ID":"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753","Type":"ContainerDied","Data":"3dad3047e8818474ff1b0e45fea437a34eb638f96303df853dd1a4ce36ee969b"} Oct 30 19:24:59.680903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:59.680812 4904 generic.go:356] "Generic (PLEG): container finished" podID="a016102a-5616-43c4-b5d5-97a01cf4774c" containerID="9d608913d9fce89172456a6ac26cdd3803de41d4181c1c5da6fd4afeffebfe51" exitCode=0 Oct 30 19:24:59.680963 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:59.680920 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" event={"ID":"a016102a-5616-43c4-b5d5-97a01cf4774c","Type":"ContainerDied","Data":"9d608913d9fce89172456a6ac26cdd3803de41d4181c1c5da6fd4afeffebfe51"} Oct 30 19:24:59.683147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:59.683034 4904 generic.go:356] "Generic (PLEG): container finished" podID="936a4b81-9973-4994-a6a9-6e7e132c35cb" containerID="c561eff633cd83ec9e9db27dee073f33466ca63292d8ae67fffc6abe499f1a46" exitCode=0 Oct 30 19:24:59.683147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:24:59.683084 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" event={"ID":"936a4b81-9973-4994-a6a9-6e7e132c35cb","Type":"ContainerDied","Data":"c561eff633cd83ec9e9db27dee073f33466ca63292d8ae67fffc6abe499f1a46"} Oct 30 19:25:00.693259 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.693218 4904 generic.go:356] "Generic (PLEG): container finished" podID="93086c4a-df9c-43ec-931d-05f425b67a01" containerID="205b25142bbbdde54dc90f9a949714e5a4b17b151fee7ff959fd64e60b725c31" exitCode=0 Oct 30 19:25:00.693646 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.693307 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" event={"ID":"93086c4a-df9c-43ec-931d-05f425b67a01","Type":"ContainerDied","Data":"205b25142bbbdde54dc90f9a949714e5a4b17b151fee7ff959fd64e60b725c31"} Oct 30 19:25:00.698219 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.698191 4904 generic.go:356] "Generic (PLEG): container finished" podID="1e17c58c-b2b2-48a9-b28b-0afad316a0d8" containerID="dd1043e98769b304980d803380a212e741eef8fac7e8f70f0e28a09c84fbda91" exitCode=0 Oct 30 19:25:00.698445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.698419 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" event={"ID":"1e17c58c-b2b2-48a9-b28b-0afad316a0d8","Type":"ContainerDied","Data":"dd1043e98769b304980d803380a212e741eef8fac7e8f70f0e28a09c84fbda91"} Oct 30 19:25:00.700874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.700847 4904 generic.go:356] "Generic (PLEG): container finished" podID="edcb47a2-a713-4645-9c7b-3e8441a28055" containerID="e1b532de7480b0360d0bce335df597bb0251564d4168c652f8f59f32979af906" exitCode=0 Oct 30 19:25:00.700968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.700952 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" event={"ID":"edcb47a2-a713-4645-9c7b-3e8441a28055","Type":"ContainerDied","Data":"e1b532de7480b0360d0bce335df597bb0251564d4168c652f8f59f32979af906"} Oct 30 19:25:00.704340 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.704223 4904 generic.go:356] "Generic (PLEG): container finished" podID="42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" containerID="3a52ff6f1eacfe206f0374d91a4698f2cd9d042bf45cd5120ed429c2304db311" exitCode=0 Oct 30 19:25:00.704340 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.704296 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" event={"ID":"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753","Type":"ContainerDied","Data":"3a52ff6f1eacfe206f0374d91a4698f2cd9d042bf45cd5120ed429c2304db311"} Oct 30 19:25:00.706774 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.706745 4904 generic.go:356] "Generic (PLEG): container finished" podID="4ce48b21-3d85-4209-bb18-47637478001f" containerID="4547b3a0facd209c3f0389fc739706628c072b870ee89997f774b811a6e2edb9" exitCode=0 Oct 30 19:25:00.706925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.706906 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" event={"ID":"4ce48b21-3d85-4209-bb18-47637478001f","Type":"ContainerDied","Data":"4547b3a0facd209c3f0389fc739706628c072b870ee89997f774b811a6e2edb9"} Oct 30 19:25:00.709671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.709648 4904 generic.go:356] "Generic (PLEG): container finished" podID="7aa41309-8972-458c-858e-678ef79c375e" containerID="ee148f37060eb72e92ce88808d70e4d68cce216d762ffa3e6959e752fed61413" exitCode=0 Oct 30 19:25:00.709764 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.709744 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" event={"ID":"7aa41309-8972-458c-858e-678ef79c375e","Type":"ContainerDied","Data":"ee148f37060eb72e92ce88808d70e4d68cce216d762ffa3e6959e752fed61413"} Oct 30 19:25:00.713566 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.712693 4904 generic.go:356] "Generic (PLEG): container finished" podID="550c652c-4e47-487a-ae4e-1324b8d44cf4" containerID="1583fb77588c33ec92c1516aecb2e0744de825ad7c0e2a8a7092d5607347c3d8" exitCode=0 Oct 30 19:25:00.713566 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.712762 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" event={"ID":"550c652c-4e47-487a-ae4e-1324b8d44cf4","Type":"ContainerDied","Data":"1583fb77588c33ec92c1516aecb2e0744de825ad7c0e2a8a7092d5607347c3d8"} Oct 30 19:25:00.715825 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.715805 4904 generic.go:356] "Generic (PLEG): container finished" podID="af15f75c-a27c-4582-b18b-7b6c9eeea6aa" containerID="e39b0f6819891824670b40bc8ca298790e36dd56bf9292e618a1bf46ec4cfe93" exitCode=0 Oct 30 19:25:00.715933 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:00.715853 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" event={"ID":"af15f75c-a27c-4582-b18b-7b6c9eeea6aa","Type":"ContainerDied","Data":"e39b0f6819891824670b40bc8ca298790e36dd56bf9292e618a1bf46ec4cfe93"} Oct 30 19:25:01.727933 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:01.727843 4904 generic.go:356] "Generic (PLEG): container finished" podID="edcb47a2-a713-4645-9c7b-3e8441a28055" containerID="07f36c498d0ccd305f7fbc548de32ce527f791dbeadefa4345f1194334dcc617" exitCode=0 Oct 30 19:25:01.728402 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:01.728012 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" event={"ID":"edcb47a2-a713-4645-9c7b-3e8441a28055","Type":"ContainerDied","Data":"07f36c498d0ccd305f7fbc548de32ce527f791dbeadefa4345f1194334dcc617"} Oct 30 19:25:01.745834 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:01.745800 4904 generic.go:356] "Generic (PLEG): container finished" podID="7aa41309-8972-458c-858e-678ef79c375e" containerID="25c1164e96dc55bbd64dfea2f0b4345428c066dc9b8e41cfa2558e8c49386614" exitCode=0 Oct 30 19:25:01.745962 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:01.745900 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" event={"ID":"7aa41309-8972-458c-858e-678ef79c375e","Type":"ContainerDied","Data":"25c1164e96dc55bbd64dfea2f0b4345428c066dc9b8e41cfa2558e8c49386614"} Oct 30 19:25:01.755951 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:01.755921 4904 generic.go:356] "Generic (PLEG): container finished" podID="91687dd6-1138-46ad-92e6-eff952029d7c" containerID="7bacf7e27b8c8e33c49f5173384121f8061d00babc122711febd40b5fba875c4" exitCode=0 Oct 30 19:25:01.756041 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:01.755975 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" event={"ID":"91687dd6-1138-46ad-92e6-eff952029d7c","Type":"ContainerDied","Data":"7bacf7e27b8c8e33c49f5173384121f8061d00babc122711febd40b5fba875c4"} Oct 30 19:25:04.812715 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:04.812672 4904 generic.go:356] "Generic (PLEG): container finished" podID="5e27b37b-33f6-4e07-b278-942cf362fe2b" containerID="1732fc50d7356a8c9868e4cf10f081f885b404e5643a35602be19fc474448c5f" exitCode=0 Oct 30 19:25:04.813125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:04.812820 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" event={"ID":"5e27b37b-33f6-4e07-b278-942cf362fe2b","Type":"ContainerDied","Data":"1732fc50d7356a8c9868e4cf10f081f885b404e5643a35602be19fc474448c5f"} Oct 30 19:25:22.063654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:22.063077 4904 generic.go:356] "Generic (PLEG): container finished" podID="936a4b81-9973-4994-a6a9-6e7e132c35cb" containerID="28aed6d56ef220c8a2a4230dda87120135b7d31b5283cae6346fb3b92efd013c" exitCode=0 Oct 30 19:25:22.063654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:22.063170 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" event={"ID":"936a4b81-9973-4994-a6a9-6e7e132c35cb","Type":"ContainerDied","Data":"28aed6d56ef220c8a2a4230dda87120135b7d31b5283cae6346fb3b92efd013c"} Oct 30 19:25:23.108889 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.108851 4904 generic.go:356] "Generic (PLEG): container finished" podID="a016102a-5616-43c4-b5d5-97a01cf4774c" containerID="cbfeb00a3c66cee5602d8cdcab05820900a166a05e5cafc94abfa298e6855464" exitCode=0 Oct 30 19:25:23.109724 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.109695 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" event={"ID":"a016102a-5616-43c4-b5d5-97a01cf4774c","Type":"ContainerDied","Data":"cbfeb00a3c66cee5602d8cdcab05820900a166a05e5cafc94abfa298e6855464"} Oct 30 19:25:23.386545 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.386468 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:25:23.465820 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.465761 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-bundle\") pod \"936a4b81-9973-4994-a6a9-6e7e132c35cb\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " Oct 30 19:25:23.465992 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.465857 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-util\" (UniqueName: \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-util\") pod \"936a4b81-9973-4994-a6a9-6e7e132c35cb\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " Oct 30 19:25:23.465992 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.465887 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936a4b81-9973-4994-a6a9-6e7e132c35cb-kube-api-access-lspzv\" (UniqueName: \"kubernetes.io/projected/936a4b81-9973-4994-a6a9-6e7e132c35cb-kube-api-access-lspzv\") pod \"936a4b81-9973-4994-a6a9-6e7e132c35cb\" (UID: \"936a4b81-9973-4994-a6a9-6e7e132c35cb\") " Oct 30 19:25:23.466623 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.466597 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-bundle" pod "936a4b81-9973-4994-a6a9-6e7e132c35cb" (UID: "936a4b81-9973-4994-a6a9-6e7e132c35cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:23.472178 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.472085 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-util" pod "936a4b81-9973-4994-a6a9-6e7e132c35cb" (UID: "936a4b81-9973-4994-a6a9-6e7e132c35cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:23.488496 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.488468 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936a4b81-9973-4994-a6a9-6e7e132c35cb-kube-api-access-lspzv" pod "936a4b81-9973-4994-a6a9-6e7e132c35cb" (UID: "936a4b81-9973-4994-a6a9-6e7e132c35cb"). InnerVolumeSpecName "kube-api-access-lspzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:23.567356 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.567315 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:23.567356 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.567351 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lspzv\" (UniqueName: \"kubernetes.io/projected/936a4b81-9973-4994-a6a9-6e7e132c35cb-kube-api-access-lspzv\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:23.567561 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:23.567367 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/936a4b81-9973-4994-a6a9-6e7e132c35cb-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:24.119589 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.119555 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" Oct 30 19:25:24.122292 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.122173 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d756c2l" event={"ID":"936a4b81-9973-4994-a6a9-6e7e132c35cb","Type":"ContainerDied","Data":"655ef768a81dc18d0c497ed73b9c9340a694e8f77d86243b9465ba996cdd55a0"} Oct 30 19:25:24.122292 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.122240 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="655ef768a81dc18d0c497ed73b9c9340a694e8f77d86243b9465ba996cdd55a0" Oct 30 19:25:24.366760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.366727 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:25:24.475477 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.475443 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-util\" (UniqueName: \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-util\") pod \"a016102a-5616-43c4-b5d5-97a01cf4774c\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " Oct 30 19:25:24.475660 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.475525 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-bundle\") pod \"a016102a-5616-43c4-b5d5-97a01cf4774c\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " Oct 30 19:25:24.475660 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.475549 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a016102a-5616-43c4-b5d5-97a01cf4774c-kube-api-access-q57b4\" (UniqueName: \"kubernetes.io/projected/a016102a-5616-43c4-b5d5-97a01cf4774c-kube-api-access-q57b4\") pod \"a016102a-5616-43c4-b5d5-97a01cf4774c\" (UID: \"a016102a-5616-43c4-b5d5-97a01cf4774c\") " Oct 30 19:25:24.476510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.476479 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-bundle" pod "a016102a-5616-43c4-b5d5-97a01cf4774c" (UID: "a016102a-5616-43c4-b5d5-97a01cf4774c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:24.488158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.481659 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-util" pod "a016102a-5616-43c4-b5d5-97a01cf4774c" (UID: "a016102a-5616-43c4-b5d5-97a01cf4774c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:24.488158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.486270 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a016102a-5616-43c4-b5d5-97a01cf4774c-kube-api-access-q57b4" pod "a016102a-5616-43c4-b5d5-97a01cf4774c" (UID: "a016102a-5616-43c4-b5d5-97a01cf4774c"). InnerVolumeSpecName "kube-api-access-q57b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:24.535568 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:25:24.535529 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:25:24.576586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.576361 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:24.576756 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.576624 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a016102a-5616-43c4-b5d5-97a01cf4774c-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:24.576756 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:24.576642 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q57b4\" (UniqueName: \"kubernetes.io/projected/a016102a-5616-43c4-b5d5-97a01cf4774c-kube-api-access-q57b4\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:25.128021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:25.127988 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" event={"ID":"a016102a-5616-43c4-b5d5-97a01cf4774c","Type":"ContainerDied","Data":"128483f0b129cffb55bd3516242efec6b50ba2260269cf53145394b95020906b"} Oct 30 19:25:25.128495 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:25.128045 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="128483f0b129cffb55bd3516242efec6b50ba2260269cf53145394b95020906b" Oct 30 19:25:25.128495 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:25.128157 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w7ql5" Oct 30 19:25:25.134490 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:25.134452 4904 generic.go:356] "Generic (PLEG): container finished" podID="1e17c58c-b2b2-48a9-b28b-0afad316a0d8" containerID="af119186b31ffc7d5093c329b9020d935acebcf94741ff05c75c59812e21fd42" exitCode=0 Oct 30 19:25:25.134632 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:25.134536 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" event={"ID":"1e17c58c-b2b2-48a9-b28b-0afad316a0d8","Type":"ContainerDied","Data":"af119186b31ffc7d5093c329b9020d935acebcf94741ff05c75c59812e21fd42"} Oct 30 19:25:26.143569 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.143535 4904 generic.go:356] "Generic (PLEG): container finished" podID="42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" containerID="6d82f3508d6c3a38b4a0a48061af82cd2836fc2d43ac8400471dc220b958bdb7" exitCode=0 Oct 30 19:25:26.143969 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.143627 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" event={"ID":"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753","Type":"ContainerDied","Data":"6d82f3508d6c3a38b4a0a48061af82cd2836fc2d43ac8400471dc220b958bdb7"} Oct 30 19:25:26.146877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.146848 4904 generic.go:356] "Generic (PLEG): container finished" podID="4ce48b21-3d85-4209-bb18-47637478001f" containerID="a046aae9d803003e6c71a7d48ce5ffedc42e70fac41b2b1b110262f02bb65a2c" exitCode=0 Oct 30 19:25:26.146995 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.146927 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" event={"ID":"4ce48b21-3d85-4209-bb18-47637478001f","Type":"ContainerDied","Data":"a046aae9d803003e6c71a7d48ce5ffedc42e70fac41b2b1b110262f02bb65a2c"} Oct 30 19:25:26.148918 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.148899 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:25:26.149692 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.149669 4904 generic.go:356] "Generic (PLEG): container finished" podID="550c652c-4e47-487a-ae4e-1324b8d44cf4" containerID="b23f3f72fe8fb6c1cec4c90bb9f6ab13298c9566c374d52f9011faea22d3234e" exitCode=1 Oct 30 19:25:26.150090 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.150067 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" event={"ID":"550c652c-4e47-487a-ae4e-1324b8d44cf4","Type":"ContainerDied","Data":"b23f3f72fe8fb6c1cec4c90bb9f6ab13298c9566c374d52f9011faea22d3234e"} Oct 30 19:25:26.314821 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.314352 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:25:26.394537 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.393888 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-util\" (UniqueName: \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-util\") pod \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " Oct 30 19:25:26.394537 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.393966 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-kube-api-access-8g5nt\" (UniqueName: \"kubernetes.io/projected/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-kube-api-access-8g5nt\") pod \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " Oct 30 19:25:26.394537 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.394047 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-bundle\") pod \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\" (UID: \"1e17c58c-b2b2-48a9-b28b-0afad316a0d8\") " Oct 30 19:25:26.395312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.395019 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-bundle" pod "1e17c58c-b2b2-48a9-b28b-0afad316a0d8" (UID: "1e17c58c-b2b2-48a9-b28b-0afad316a0d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:26.397360 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.397196 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-kube-api-access-8g5nt" pod "1e17c58c-b2b2-48a9-b28b-0afad316a0d8" (UID: "1e17c58c-b2b2-48a9-b28b-0afad316a0d8"). InnerVolumeSpecName "kube-api-access-8g5nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:26.402479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.402453 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-util" pod "1e17c58c-b2b2-48a9-b28b-0afad316a0d8" (UID: "1e17c58c-b2b2-48a9-b28b-0afad316a0d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:26.495469 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.495441 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:26.495469 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.495471 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g5nt\" (UniqueName: \"kubernetes.io/projected/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-kube-api-access-8g5nt\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:26.495697 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:26.495511 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e17c58c-b2b2-48a9-b28b-0afad316a0d8-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:27.156268 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.156236 4904 generic.go:356] "Generic (PLEG): container finished" podID="93086c4a-df9c-43ec-931d-05f425b67a01" containerID="6f21853c5ffef3249fcfb28833396c743e83b9025e36f00395c3cae3ce07ce00" exitCode=0 Oct 30 19:25:27.156681 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.156320 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" event={"ID":"93086c4a-df9c-43ec-931d-05f425b67a01","Type":"ContainerDied","Data":"6f21853c5ffef3249fcfb28833396c743e83b9025e36f00395c3cae3ce07ce00"} Oct 30 19:25:27.158188 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.158166 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" Oct 30 19:25:27.158327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.158167 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h48fm" event={"ID":"1e17c58c-b2b2-48a9-b28b-0afad316a0d8","Type":"ContainerDied","Data":"53c5a3dea3c80771d4603398fc3e889e56b220b77d59bf5f05108ed53a427a7d"} Oct 30 19:25:27.158327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.158289 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53c5a3dea3c80771d4603398fc3e889e56b220b77d59bf5f05108ed53a427a7d" Oct 30 19:25:27.160242 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.160213 4904 generic.go:356] "Generic (PLEG): container finished" podID="edcb47a2-a713-4645-9c7b-3e8441a28055" containerID="81c53d067ba7006efa44d355d0eeee96518ace1fa810ab3d1697f4233285c13a" exitCode=0 Oct 30 19:25:27.160351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.160247 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" event={"ID":"edcb47a2-a713-4645-9c7b-3e8441a28055","Type":"ContainerDied","Data":"81c53d067ba7006efa44d355d0eeee96518ace1fa810ab3d1697f4233285c13a"} Oct 30 19:25:27.162484 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.162462 4904 generic.go:356] "Generic (PLEG): container finished" podID="7aa41309-8972-458c-858e-678ef79c375e" containerID="801ec3af475cbd8c62c9dce49c32cd473e9f02e0fd80ccc414f01f1f434d873b" exitCode=0 Oct 30 19:25:27.162599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.162542 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" event={"ID":"7aa41309-8972-458c-858e-678ef79c375e","Type":"ContainerDied","Data":"801ec3af475cbd8c62c9dce49c32cd473e9f02e0fd80ccc414f01f1f434d873b"} Oct 30 19:25:27.164717 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.164694 4904 generic.go:356] "Generic (PLEG): container finished" podID="91687dd6-1138-46ad-92e6-eff952029d7c" containerID="e1a216dee4ec4d4f494cb544e2e3574ffad23ee7646a2de06f0652ec475ddae8" exitCode=0 Oct 30 19:25:27.164838 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.164763 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" event={"ID":"91687dd6-1138-46ad-92e6-eff952029d7c","Type":"ContainerDied","Data":"e1a216dee4ec4d4f494cb544e2e3574ffad23ee7646a2de06f0652ec475ddae8"} Oct 30 19:25:27.167057 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.166788 4904 generic.go:356] "Generic (PLEG): container finished" podID="af15f75c-a27c-4582-b18b-7b6c9eeea6aa" containerID="16a471b3b0e11f4905c2e7a6a884be9c2232f7f1cdaabbbc40e25b1f31ff8ee4" exitCode=0 Oct 30 19:25:27.167057 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.166895 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" event={"ID":"af15f75c-a27c-4582-b18b-7b6c9eeea6aa","Type":"ContainerDied","Data":"16a471b3b0e11f4905c2e7a6a884be9c2232f7f1cdaabbbc40e25b1f31ff8ee4"} Oct 30 19:25:27.333689 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.333668 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:25:27.374319 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.374293 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:25:27.374495 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.374475 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:25:27.377084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.377067 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:25:27.404707 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.404688 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-bundle\" (UniqueName: \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-bundle\") pod \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " Oct 30 19:25:27.404813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.404721 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-kube-api-access-8jwqs\" (UniqueName: \"kubernetes.io/projected/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-kube-api-access-8jwqs\") pod \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " Oct 30 19:25:27.404813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.404750 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-util\" (UniqueName: \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-util\") pod \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\" (UID: \"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753\") " Oct 30 19:25:27.405394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.405374 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-bundle" pod "42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" (UID: "42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:27.406885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.406839 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-kube-api-access-8jwqs" pod "42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" (UID: "42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753"). InnerVolumeSpecName "kube-api-access-8jwqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:27.410145 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.410124 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-util" pod "42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" (UID: "42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:27.506243 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.506214 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-util\" (UniqueName: \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-util\") pod \"550c652c-4e47-487a-ae4e-1324b8d44cf4\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " Oct 30 19:25:27.506352 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.506318 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-bundle\") pod \"4ce48b21-3d85-4209-bb18-47637478001f\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " Oct 30 19:25:27.506416 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.506372 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce48b21-3d85-4209-bb18-47637478001f-kube-api-access-bhxtx\" (UniqueName: \"kubernetes.io/projected/4ce48b21-3d85-4209-bb18-47637478001f-kube-api-access-bhxtx\") pod \"4ce48b21-3d85-4209-bb18-47637478001f\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " Oct 30 19:25:27.506478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.506444 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-bundle\") pod \"550c652c-4e47-487a-ae4e-1324b8d44cf4\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " Oct 30 19:25:27.506478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.506473 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550c652c-4e47-487a-ae4e-1324b8d44cf4-kube-api-access-l7nxc\" (UniqueName: \"kubernetes.io/projected/550c652c-4e47-487a-ae4e-1324b8d44cf4-kube-api-access-l7nxc\") pod \"550c652c-4e47-487a-ae4e-1324b8d44cf4\" (UID: \"550c652c-4e47-487a-ae4e-1324b8d44cf4\") " Oct 30 19:25:27.506587 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.506503 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-util\" (UniqueName: \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-util\") pod \"4ce48b21-3d85-4209-bb18-47637478001f\" (UID: \"4ce48b21-3d85-4209-bb18-47637478001f\") " Oct 30 19:25:27.506781 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.506760 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:27.506922 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.506908 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:27.507032 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.507019 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jwqs\" (UniqueName: \"kubernetes.io/projected/42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753-kube-api-access-8jwqs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:27.507275 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.507248 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-bundle" pod "4ce48b21-3d85-4209-bb18-47637478001f" (UID: "4ce48b21-3d85-4209-bb18-47637478001f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:27.507601 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.507575 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-bundle" pod "550c652c-4e47-487a-ae4e-1324b8d44cf4" (UID: "550c652c-4e47-487a-ae4e-1324b8d44cf4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:27.508728 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.508704 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce48b21-3d85-4209-bb18-47637478001f-kube-api-access-bhxtx" pod "4ce48b21-3d85-4209-bb18-47637478001f" (UID: "4ce48b21-3d85-4209-bb18-47637478001f"). InnerVolumeSpecName "kube-api-access-bhxtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:27.509012 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.508992 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550c652c-4e47-487a-ae4e-1324b8d44cf4-kube-api-access-l7nxc" pod "550c652c-4e47-487a-ae4e-1324b8d44cf4" (UID: "550c652c-4e47-487a-ae4e-1324b8d44cf4"). InnerVolumeSpecName "kube-api-access-l7nxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:27.513024 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.512997 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-util" pod "550c652c-4e47-487a-ae4e-1324b8d44cf4" (UID: "550c652c-4e47-487a-ae4e-1324b8d44cf4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:27.513146 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.513131 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-util" pod "4ce48b21-3d85-4209-bb18-47637478001f" (UID: "4ce48b21-3d85-4209-bb18-47637478001f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:27.608117 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.608079 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:27.608209 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.608121 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:27.608209 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.608130 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhxtx\" (UniqueName: \"kubernetes.io/projected/4ce48b21-3d85-4209-bb18-47637478001f-kube-api-access-bhxtx\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:27.608209 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.608140 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/550c652c-4e47-487a-ae4e-1324b8d44cf4-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:27.608209 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.608148 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nxc\" (UniqueName: \"kubernetes.io/projected/550c652c-4e47-487a-ae4e-1324b8d44cf4-kube-api-access-l7nxc\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:27.608209 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:27.608155 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ce48b21-3d85-4209-bb18-47637478001f-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.174176 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.174152 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:25:28.175057 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.175035 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" Oct 30 19:25:28.175057 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.175043 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd" event={"ID":"550c652c-4e47-487a-ae4e-1324b8d44cf4","Type":"ContainerDied","Data":"005f17fa92031d59918432c1b5898eb2fbc6fa443c0e6c958e9a04b4469f3647"} Oct 30 19:25:28.175229 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.175116 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="005f17fa92031d59918432c1b5898eb2fbc6fa443c0e6c958e9a04b4469f3647" Oct 30 19:25:28.177412 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.177390 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" event={"ID":"42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753","Type":"ContainerDied","Data":"1a90fdd0fa37ac463aa045cbd3d5877aec95c102682682c55c81e059a98d57c7"} Oct 30 19:25:28.177412 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.177414 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am222n" Oct 30 19:25:28.177579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.177431 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a90fdd0fa37ac463aa045cbd3d5877aec95c102682682c55c81e059a98d57c7" Oct 30 19:25:28.179713 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.179688 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" Oct 30 19:25:28.179834 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.179691 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gn52q" event={"ID":"4ce48b21-3d85-4209-bb18-47637478001f","Type":"ContainerDied","Data":"12d1cb0a0657d756c1815c44cf99f650531ab87e1268b9ed8fbd489b763a7155"} Oct 30 19:25:28.179897 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.179827 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12d1cb0a0657d756c1815c44cf99f650531ab87e1268b9ed8fbd489b763a7155" Oct 30 19:25:28.182169 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.182142 4904 generic.go:356] "Generic (PLEG): container finished" podID="5e27b37b-33f6-4e07-b278-942cf362fe2b" containerID="1e54618f2ac2479f7bfc175adbffe4bb234b397e65d48b7e40b33278b7712355" exitCode=0 Oct 30 19:25:28.182272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.182233 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" event={"ID":"5e27b37b-33f6-4e07-b278-942cf362fe2b","Type":"ContainerDied","Data":"1e54618f2ac2479f7bfc175adbffe4bb234b397e65d48b7e40b33278b7712355"} Oct 30 19:25:28.352674 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.352651 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:25:28.412454 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.412425 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:25:28.414210 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.414186 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91687dd6-1138-46ad-92e6-eff952029d7c-kube-api-access-cp5t2\" (UniqueName: \"kubernetes.io/projected/91687dd6-1138-46ad-92e6-eff952029d7c-kube-api-access-cp5t2\") pod \"91687dd6-1138-46ad-92e6-eff952029d7c\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " Oct 30 19:25:28.414315 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.414278 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-bundle\") pod \"91687dd6-1138-46ad-92e6-eff952029d7c\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " Oct 30 19:25:28.414388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.414316 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-util\" (UniqueName: \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-util\") pod \"91687dd6-1138-46ad-92e6-eff952029d7c\" (UID: \"91687dd6-1138-46ad-92e6-eff952029d7c\") " Oct 30 19:25:28.414813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.414793 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-bundle" pod "91687dd6-1138-46ad-92e6-eff952029d7c" (UID: "91687dd6-1138-46ad-92e6-eff952029d7c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:28.415934 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.415907 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:25:28.416783 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.416758 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91687dd6-1138-46ad-92e6-eff952029d7c-kube-api-access-cp5t2" pod "91687dd6-1138-46ad-92e6-eff952029d7c" (UID: "91687dd6-1138-46ad-92e6-eff952029d7c"). InnerVolumeSpecName "kube-api-access-cp5t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:28.419838 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.419804 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:25:28.421650 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.421629 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-util" pod "91687dd6-1138-46ad-92e6-eff952029d7c" (UID: "91687dd6-1138-46ad-92e6-eff952029d7c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:28.422943 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.422924 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:25:28.515146 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515039 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-bundle\" (UniqueName: \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-bundle\") pod \"edcb47a2-a713-4645-9c7b-3e8441a28055\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " Oct 30 19:25:28.515292 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515157 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93086c4a-df9c-43ec-931d-05f425b67a01-kube-api-access-8r98x\" (UniqueName: \"kubernetes.io/projected/93086c4a-df9c-43ec-931d-05f425b67a01-kube-api-access-8r98x\") pod \"93086c4a-df9c-43ec-931d-05f425b67a01\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " Oct 30 19:25:28.515292 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515222 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-util\" (UniqueName: \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-util\") pod \"7aa41309-8972-458c-858e-678ef79c375e\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " Oct 30 19:25:28.515292 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515262 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-kube-api-access-8sj8h\" (UniqueName: \"kubernetes.io/projected/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-kube-api-access-8sj8h\") pod \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " Oct 30 19:25:28.515448 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515331 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-util\" (UniqueName: \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-util\") pod \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " Oct 30 19:25:28.515448 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515371 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-util\" (UniqueName: \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-util\") pod \"edcb47a2-a713-4645-9c7b-3e8441a28055\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " Oct 30 19:25:28.515448 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515409 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-util\" (UniqueName: \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-util\") pod \"93086c4a-df9c-43ec-931d-05f425b67a01\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " Oct 30 19:25:28.515600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515455 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa41309-8972-458c-858e-678ef79c375e-kube-api-access-x524q\" (UniqueName: \"kubernetes.io/projected/7aa41309-8972-458c-858e-678ef79c375e-kube-api-access-x524q\") pod \"7aa41309-8972-458c-858e-678ef79c375e\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " Oct 30 19:25:28.515600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515498 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-bundle\") pod \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\" (UID: \"af15f75c-a27c-4582-b18b-7b6c9eeea6aa\") " Oct 30 19:25:28.515600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515570 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcb47a2-a713-4645-9c7b-3e8441a28055-kube-api-access-qh2mk\" (UniqueName: \"kubernetes.io/projected/edcb47a2-a713-4645-9c7b-3e8441a28055-kube-api-access-qh2mk\") pod \"edcb47a2-a713-4645-9c7b-3e8441a28055\" (UID: \"edcb47a2-a713-4645-9c7b-3e8441a28055\") " Oct 30 19:25:28.515744 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515645 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-bundle\" (UniqueName: \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-bundle\") pod \"93086c4a-df9c-43ec-931d-05f425b67a01\" (UID: \"93086c4a-df9c-43ec-931d-05f425b67a01\") " Oct 30 19:25:28.515744 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515686 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-bundle\") pod \"7aa41309-8972-458c-858e-678ef79c375e\" (UID: \"7aa41309-8972-458c-858e-678ef79c375e\") " Oct 30 19:25:28.515918 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515898 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.515978 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515926 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp5t2\" (UniqueName: \"kubernetes.io/projected/91687dd6-1138-46ad-92e6-eff952029d7c-kube-api-access-cp5t2\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.515978 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.515941 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91687dd6-1138-46ad-92e6-eff952029d7c-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.517143 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.517086 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-bundle" pod "7aa41309-8972-458c-858e-678ef79c375e" (UID: "7aa41309-8972-458c-858e-678ef79c375e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:28.518020 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.517994 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-bundle" pod "edcb47a2-a713-4645-9c7b-3e8441a28055" (UID: "edcb47a2-a713-4645-9c7b-3e8441a28055"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:28.518322 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.518296 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa41309-8972-458c-858e-678ef79c375e-kube-api-access-x524q" pod "7aa41309-8972-458c-858e-678ef79c375e" (UID: "7aa41309-8972-458c-858e-678ef79c375e"). InnerVolumeSpecName "kube-api-access-x524q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:28.518449 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.518419 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-kube-api-access-8sj8h" pod "af15f75c-a27c-4582-b18b-7b6c9eeea6aa" (UID: "af15f75c-a27c-4582-b18b-7b6c9eeea6aa"). InnerVolumeSpecName "kube-api-access-8sj8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:28.518797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.518537 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93086c4a-df9c-43ec-931d-05f425b67a01-kube-api-access-8r98x" pod "93086c4a-df9c-43ec-931d-05f425b67a01" (UID: "93086c4a-df9c-43ec-931d-05f425b67a01"). InnerVolumeSpecName "kube-api-access-8r98x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:28.518797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.518754 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcb47a2-a713-4645-9c7b-3e8441a28055-kube-api-access-qh2mk" pod "edcb47a2-a713-4645-9c7b-3e8441a28055" (UID: "edcb47a2-a713-4645-9c7b-3e8441a28055"). InnerVolumeSpecName "kube-api-access-qh2mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:28.519010 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.518984 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-bundle" pod "af15f75c-a27c-4582-b18b-7b6c9eeea6aa" (UID: "af15f75c-a27c-4582-b18b-7b6c9eeea6aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:28.519575 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.519543 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-bundle" pod "93086c4a-df9c-43ec-931d-05f425b67a01" (UID: "93086c4a-df9c-43ec-931d-05f425b67a01"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:28.524733 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.524706 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-util" pod "edcb47a2-a713-4645-9c7b-3e8441a28055" (UID: "edcb47a2-a713-4645-9c7b-3e8441a28055"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:28.525519 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.525491 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-util" pod "7aa41309-8972-458c-858e-678ef79c375e" (UID: "7aa41309-8972-458c-858e-678ef79c375e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:28.527569 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.527547 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-util" pod "93086c4a-df9c-43ec-931d-05f425b67a01" (UID: "93086c4a-df9c-43ec-931d-05f425b67a01"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:28.527974 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.527945 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-util" pod "af15f75c-a27c-4582-b18b-7b6c9eeea6aa" (UID: "af15f75c-a27c-4582-b18b-7b6c9eeea6aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:28.616401 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616377 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616401 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616396 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r98x\" (UniqueName: \"kubernetes.io/projected/93086c4a-df9c-43ec-931d-05f425b67a01-kube-api-access-8r98x\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616526 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616406 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616526 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616414 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sj8h\" (UniqueName: \"kubernetes.io/projected/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-kube-api-access-8sj8h\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616526 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616422 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616526 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616429 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edcb47a2-a713-4645-9c7b-3e8441a28055-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616526 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616445 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616526 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616452 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x524q\" (UniqueName: \"kubernetes.io/projected/7aa41309-8972-458c-858e-678ef79c375e-kube-api-access-x524q\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616526 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616461 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af15f75c-a27c-4582-b18b-7b6c9eeea6aa-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616526 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616470 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh2mk\" (UniqueName: \"kubernetes.io/projected/edcb47a2-a713-4645-9c7b-3e8441a28055-kube-api-access-qh2mk\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616526 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616478 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93086c4a-df9c-43ec-931d-05f425b67a01-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:28.616526 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:28.616486 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa41309-8972-458c-858e-678ef79c375e-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:29.189527 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.189498 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" Oct 30 19:25:29.189945 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.189494 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhpg6" event={"ID":"7aa41309-8972-458c-858e-678ef79c375e","Type":"ContainerDied","Data":"3e6ecca7d081f9327b2760115685e0ab50a14a36ae778373e92e85827ea7ca69"} Oct 30 19:25:29.189945 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.189673 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6ecca7d081f9327b2760115685e0ab50a14a36ae778373e92e85827ea7ca69" Oct 30 19:25:29.191989 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.191969 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" Oct 30 19:25:29.191989 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.191973 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6n499" event={"ID":"91687dd6-1138-46ad-92e6-eff952029d7c","Type":"ContainerDied","Data":"c637137adcaf8ec3e4277e04c6a566deeafdef819b9ebca1c30fdd6ad0a9368c"} Oct 30 19:25:29.192210 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.192015 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c637137adcaf8ec3e4277e04c6a566deeafdef819b9ebca1c30fdd6ad0a9368c" Oct 30 19:25:29.194325 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.194299 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" event={"ID":"af15f75c-a27c-4582-b18b-7b6c9eeea6aa","Type":"ContainerDied","Data":"6d861e6b4d1f09c5471ce9f91c590262b5ad78ee0a7545ad73b20d86fbdf66aa"} Oct 30 19:25:29.194325 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.194317 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j8zpr" Oct 30 19:25:29.194477 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.194336 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d861e6b4d1f09c5471ce9f91c590262b5ad78ee0a7545ad73b20d86fbdf66aa" Oct 30 19:25:29.196767 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.196737 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" event={"ID":"93086c4a-df9c-43ec-931d-05f425b67a01","Type":"ContainerDied","Data":"e1143985368fff477cacb45d6c743f40bceccce705b7a3c0cf414a15ca118f06"} Oct 30 19:25:29.196876 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.196784 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6sc5" Oct 30 19:25:29.196876 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.196791 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1143985368fff477cacb45d6c743f40bceccce705b7a3c0cf414a15ca118f06" Oct 30 19:25:29.199021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.198996 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" Oct 30 19:25:29.199202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.199176 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjmtdh" event={"ID":"edcb47a2-a713-4645-9c7b-3e8441a28055","Type":"ContainerDied","Data":"90ce99e5ed92eca3650af918fc7f2c10c3420664559f47f3a1b500b594d70461"} Oct 30 19:25:29.199293 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.199228 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90ce99e5ed92eca3650af918fc7f2c10c3420664559f47f3a1b500b594d70461" Oct 30 19:25:29.339587 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.339562 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:25:29.420660 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.420637 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e27b37b-33f6-4e07-b278-942cf362fe2b-kube-api-access-qxnmh\" (UniqueName: \"kubernetes.io/projected/5e27b37b-33f6-4e07-b278-942cf362fe2b-kube-api-access-qxnmh\") pod \"5e27b37b-33f6-4e07-b278-942cf362fe2b\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " Oct 30 19:25:29.420778 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.420672 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-bundle\") pod \"5e27b37b-33f6-4e07-b278-942cf362fe2b\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " Oct 30 19:25:29.420778 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.420719 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-util\" (UniqueName: \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-util\") pod \"5e27b37b-33f6-4e07-b278-942cf362fe2b\" (UID: \"5e27b37b-33f6-4e07-b278-942cf362fe2b\") " Oct 30 19:25:29.421060 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.421038 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-bundle" pod "5e27b37b-33f6-4e07-b278-942cf362fe2b" (UID: "5e27b37b-33f6-4e07-b278-942cf362fe2b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:29.422815 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.422786 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e27b37b-33f6-4e07-b278-942cf362fe2b-kube-api-access-qxnmh" pod "5e27b37b-33f6-4e07-b278-942cf362fe2b" (UID: "5e27b37b-33f6-4e07-b278-942cf362fe2b"). InnerVolumeSpecName "kube-api-access-qxnmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:29.427397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.427377 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-util" pod "5e27b37b-33f6-4e07-b278-942cf362fe2b" (UID: "5e27b37b-33f6-4e07-b278-942cf362fe2b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:29.521505 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.521445 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:29.521505 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.521469 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxnmh\" (UniqueName: \"kubernetes.io/projected/5e27b37b-33f6-4e07-b278-942cf362fe2b-kube-api-access-qxnmh\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:29.521505 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:29.521480 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e27b37b-33f6-4e07-b278-942cf362fe2b-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:30.207772 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:30.207736 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" event={"ID":"5e27b37b-33f6-4e07-b278-942cf362fe2b","Type":"ContainerDied","Data":"36313f808d4a78bd7c5f4b3612ad610afff1be4060d144376349f194ed910f27"} Oct 30 19:25:30.208190 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:30.207779 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bqpsf" Oct 30 19:25:30.208190 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:30.207786 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36313f808d4a78bd7c5f4b3612ad610afff1be4060d144376349f194ed910f27" Oct 30 19:25:32.475124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475077 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv"] Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475380 4904 cpu_manager.go:414] "containerMap: removing container" podUID="93086c4a-df9c-43ec-931d-05f425b67a01" containerName="util" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475391 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93086c4a-df9c-43ec-931d-05f425b67a01" containerName="util" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475401 4904 cpu_manager.go:414] "containerMap: removing container" podUID="5e27b37b-33f6-4e07-b278-942cf362fe2b" containerName="extract" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475408 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e27b37b-33f6-4e07-b278-942cf362fe2b" containerName="extract" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475417 4904 cpu_manager.go:414] "containerMap: removing container" podUID="af15f75c-a27c-4582-b18b-7b6c9eeea6aa" containerName="util" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475421 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af15f75c-a27c-4582-b18b-7b6c9eeea6aa" containerName="util" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475427 4904 cpu_manager.go:414] "containerMap: removing container" podUID="5e27b37b-33f6-4e07-b278-942cf362fe2b" containerName="util" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475432 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e27b37b-33f6-4e07-b278-942cf362fe2b" containerName="util" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475438 4904 cpu_manager.go:414] "containerMap: removing container" podUID="7aa41309-8972-458c-858e-678ef79c375e" containerName="pull" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475443 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa41309-8972-458c-858e-678ef79c375e" containerName="pull" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475452 4904 cpu_manager.go:414] "containerMap: removing container" podUID="91687dd6-1138-46ad-92e6-eff952029d7c" containerName="util" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475457 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91687dd6-1138-46ad-92e6-eff952029d7c" containerName="util" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475463 4904 cpu_manager.go:414] "containerMap: removing container" podUID="af15f75c-a27c-4582-b18b-7b6c9eeea6aa" containerName="pull" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475467 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af15f75c-a27c-4582-b18b-7b6c9eeea6aa" containerName="pull" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475473 4904 cpu_manager.go:414] "containerMap: removing container" podUID="550c652c-4e47-487a-ae4e-1324b8d44cf4" containerName="pull" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475477 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550c652c-4e47-487a-ae4e-1324b8d44cf4" containerName="pull" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475483 4904 cpu_manager.go:414] "containerMap: removing container" podUID="4ce48b21-3d85-4209-bb18-47637478001f" containerName="extract" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475487 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce48b21-3d85-4209-bb18-47637478001f" containerName="extract" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475492 4904 cpu_manager.go:414] "containerMap: removing container" podUID="7aa41309-8972-458c-858e-678ef79c375e" containerName="extract" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475497 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa41309-8972-458c-858e-678ef79c375e" containerName="extract" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475506 4904 cpu_manager.go:414] "containerMap: removing container" podUID="af15f75c-a27c-4582-b18b-7b6c9eeea6aa" containerName="extract" Oct 30 19:25:32.475497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475510 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af15f75c-a27c-4582-b18b-7b6c9eeea6aa" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475520 4904 cpu_manager.go:414] "containerMap: removing container" podUID="a016102a-5616-43c4-b5d5-97a01cf4774c" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475524 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a016102a-5616-43c4-b5d5-97a01cf4774c" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475529 4904 cpu_manager.go:414] "containerMap: removing container" podUID="42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475535 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475540 4904 cpu_manager.go:414] "containerMap: removing container" podUID="1e17c58c-b2b2-48a9-b28b-0afad316a0d8" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475545 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e17c58c-b2b2-48a9-b28b-0afad316a0d8" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475549 4904 cpu_manager.go:414] "containerMap: removing container" podUID="5e27b37b-33f6-4e07-b278-942cf362fe2b" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475554 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e27b37b-33f6-4e07-b278-942cf362fe2b" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475561 4904 cpu_manager.go:414] "containerMap: removing container" podUID="1e17c58c-b2b2-48a9-b28b-0afad316a0d8" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475564 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e17c58c-b2b2-48a9-b28b-0afad316a0d8" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475570 4904 cpu_manager.go:414] "containerMap: removing container" podUID="42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475573 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475579 4904 cpu_manager.go:414] "containerMap: removing container" podUID="936a4b81-9973-4994-a6a9-6e7e132c35cb" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475583 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936a4b81-9973-4994-a6a9-6e7e132c35cb" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475591 4904 cpu_manager.go:414] "containerMap: removing container" podUID="4ce48b21-3d85-4209-bb18-47637478001f" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475596 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce48b21-3d85-4209-bb18-47637478001f" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475603 4904 cpu_manager.go:414] "containerMap: removing container" podUID="4ce48b21-3d85-4209-bb18-47637478001f" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475610 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce48b21-3d85-4209-bb18-47637478001f" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475618 4904 cpu_manager.go:414] "containerMap: removing container" podUID="edcb47a2-a713-4645-9c7b-3e8441a28055" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475624 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcb47a2-a713-4645-9c7b-3e8441a28055" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475632 4904 cpu_manager.go:414] "containerMap: removing container" podUID="91687dd6-1138-46ad-92e6-eff952029d7c" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475639 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91687dd6-1138-46ad-92e6-eff952029d7c" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475659 4904 cpu_manager.go:414] "containerMap: removing container" podUID="42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475666 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475679 4904 cpu_manager.go:414] "containerMap: removing container" podUID="93086c4a-df9c-43ec-931d-05f425b67a01" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475685 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93086c4a-df9c-43ec-931d-05f425b67a01" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475694 4904 cpu_manager.go:414] "containerMap: removing container" podUID="edcb47a2-a713-4645-9c7b-3e8441a28055" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475699 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcb47a2-a713-4645-9c7b-3e8441a28055" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475705 4904 cpu_manager.go:414] "containerMap: removing container" podUID="a016102a-5616-43c4-b5d5-97a01cf4774c" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475709 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a016102a-5616-43c4-b5d5-97a01cf4774c" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475714 4904 cpu_manager.go:414] "containerMap: removing container" podUID="1e17c58c-b2b2-48a9-b28b-0afad316a0d8" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475718 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e17c58c-b2b2-48a9-b28b-0afad316a0d8" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475725 4904 cpu_manager.go:414] "containerMap: removing container" podUID="550c652c-4e47-487a-ae4e-1324b8d44cf4" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475729 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550c652c-4e47-487a-ae4e-1324b8d44cf4" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475737 4904 cpu_manager.go:414] "containerMap: removing container" podUID="936a4b81-9973-4994-a6a9-6e7e132c35cb" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475741 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936a4b81-9973-4994-a6a9-6e7e132c35cb" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475749 4904 cpu_manager.go:414] "containerMap: removing container" podUID="edcb47a2-a713-4645-9c7b-3e8441a28055" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475753 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcb47a2-a713-4645-9c7b-3e8441a28055" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475762 4904 cpu_manager.go:414] "containerMap: removing container" podUID="7aa41309-8972-458c-858e-678ef79c375e" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475766 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa41309-8972-458c-858e-678ef79c375e" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475772 4904 cpu_manager.go:414] "containerMap: removing container" podUID="91687dd6-1138-46ad-92e6-eff952029d7c" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475776 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91687dd6-1138-46ad-92e6-eff952029d7c" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475782 4904 cpu_manager.go:414] "containerMap: removing container" podUID="550c652c-4e47-487a-ae4e-1324b8d44cf4" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475786 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550c652c-4e47-487a-ae4e-1324b8d44cf4" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475792 4904 cpu_manager.go:414] "containerMap: removing container" podUID="936a4b81-9973-4994-a6a9-6e7e132c35cb" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475795 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936a4b81-9973-4994-a6a9-6e7e132c35cb" containerName="util" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475800 4904 cpu_manager.go:414] "containerMap: removing container" podUID="a016102a-5616-43c4-b5d5-97a01cf4774c" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475804 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a016102a-5616-43c4-b5d5-97a01cf4774c" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475810 4904 cpu_manager.go:414] "containerMap: removing container" podUID="93086c4a-df9c-43ec-931d-05f425b67a01" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475814 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93086c4a-df9c-43ec-931d-05f425b67a01" containerName="pull" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475877 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa41309-8972-458c-858e-678ef79c375e" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475888 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ce48b21-3d85-4209-bb18-47637478001f" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475898 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="af15f75c-a27c-4582-b18b-7b6c9eeea6aa" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475905 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="936a4b81-9973-4994-a6a9-6e7e132c35cb" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475912 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fe3bf8-5c19-4bac-ad0c-1d31d2dc1753" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475918 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="93086c4a-df9c-43ec-931d-05f425b67a01" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475923 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="a016102a-5616-43c4-b5d5-97a01cf4774c" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475928 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e17c58c-b2b2-48a9-b28b-0afad316a0d8" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475933 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="550c652c-4e47-487a-ae4e-1324b8d44cf4" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475938 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="edcb47a2-a713-4645-9c7b-3e8441a28055" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475943 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="91687dd6-1138-46ad-92e6-eff952029d7c" containerName="extract" Oct 30 19:25:32.476163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.475948 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e27b37b-33f6-4e07-b278-942cf362fe2b" containerName="extract" Oct 30 19:25:32.498011 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.497990 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.529251 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.529228 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 19:25:32.529355 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.529262 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:25:32.529462 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.529447 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 19:25:32.529536 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.529520 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8hq8l\"" Oct 30 19:25:32.529592 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.529514 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:25:32.529654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.529636 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:25:32.584174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.584147 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv"] Oct 30 19:25:32.643706 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.643678 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ab94ee3-20b2-46e8-84a7-285824abb7ae-manifests\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.643833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.643711 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ab94ee3-20b2-46e8-84a7-285824abb7ae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.643833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.643749 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab94ee3-20b2-46e8-84a7-285824abb7ae-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.643942 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.643859 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ab94ee3-20b2-46e8-84a7-285824abb7ae-tmp\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.643942 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.643882 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5p2\" (UniqueName: \"kubernetes.io/projected/5ab94ee3-20b2-46e8-84a7-285824abb7ae-kube-api-access-sq5p2\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.643942 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.643915 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ab94ee3-20b2-46e8-84a7-285824abb7ae-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.744865 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.744793 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ab94ee3-20b2-46e8-84a7-285824abb7ae-manifests\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.744865 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.744834 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ab94ee3-20b2-46e8-84a7-285824abb7ae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.745044 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.744892 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab94ee3-20b2-46e8-84a7-285824abb7ae-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.745044 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.744933 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ab94ee3-20b2-46e8-84a7-285824abb7ae-tmp\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.745044 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.744957 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5p2\" (UniqueName: \"kubernetes.io/projected/5ab94ee3-20b2-46e8-84a7-285824abb7ae-kube-api-access-sq5p2\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.745044 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.744999 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ab94ee3-20b2-46e8-84a7-285824abb7ae-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.745271 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.745252 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ab94ee3-20b2-46e8-84a7-285824abb7ae-manifests\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.747541 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.747515 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ab94ee3-20b2-46e8-84a7-285824abb7ae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.747635 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.747558 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab94ee3-20b2-46e8-84a7-285824abb7ae-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.747686 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.747666 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ab94ee3-20b2-46e8-84a7-285824abb7ae-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.747748 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.747733 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ab94ee3-20b2-46e8-84a7-285824abb7ae-tmp\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.767271 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.767250 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5p2\" (UniqueName: \"kubernetes.io/projected/5ab94ee3-20b2-46e8-84a7-285824abb7ae-kube-api-access-sq5p2\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-ngmkv\" (UID: \"5ab94ee3-20b2-46e8-84a7-285824abb7ae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:32.842814 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:32.842796 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:33.253472 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:33.253446 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv"] Oct 30 19:25:33.257474 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:25:33.257450 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab94ee3_20b2_46e8_84a7_285824abb7ae.slice/crio-f4a5172c4c202036f29fca1a88f675f08bbd4a3460fe032d19e7584d24373530 WatchSource:0}: Error finding container f4a5172c4c202036f29fca1a88f675f08bbd4a3460fe032d19e7584d24373530: Status 404 returned error can't find the container with id f4a5172c4c202036f29fca1a88f675f08bbd4a3460fe032d19e7584d24373530 Oct 30 19:25:34.230647 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:34.230607 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" event={"ID":"5ab94ee3-20b2-46e8-84a7-285824abb7ae","Type":"ContainerStarted","Data":"f4a5172c4c202036f29fca1a88f675f08bbd4a3460fe032d19e7584d24373530"} Oct 30 19:25:36.041508 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.041485 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq"] Oct 30 19:25:36.045032 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.045016 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.062515 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.062495 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq"] Oct 30 19:25:36.073575 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.073555 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:25:36.073838 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.073815 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:25:36.073838 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.073830 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-rltxw\"" Oct 30 19:25:36.169466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.169441 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.169557 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.169486 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5n7\" (UniqueName: \"kubernetes.io/projected/2b71c1f3-ad2b-4aef-8256-693b02b702af-kube-api-access-rl5n7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.169557 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.169513 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.241793 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.241729 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" event={"ID":"5ab94ee3-20b2-46e8-84a7-285824abb7ae","Type":"ContainerStarted","Data":"508fcc77464e5aea14fa7aef74390f118bfe68327d851fd4ad364eb28891d30d"} Oct 30 19:25:36.270738 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.270720 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.270824 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.270752 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5n7\" (UniqueName: \"kubernetes.io/projected/2b71c1f3-ad2b-4aef-8256-693b02b702af-kube-api-access-rl5n7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.270824 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.270773 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.271023 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.271005 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.271075 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.271037 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.291316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.291295 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5n7\" (UniqueName: \"kubernetes.io/projected/2b71c1f3-ad2b-4aef-8256-693b02b702af-kube-api-access-rl5n7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.360347 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.360328 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:36.662437 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:36.662415 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq"] Oct 30 19:25:36.682894 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:25:36.682864 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b71c1f3_ad2b_4aef_8256_693b02b702af.slice/crio-12fee6f35b14bc15b5fafdccbebc36cf66bc3d255930b93931d451514b65cf00 WatchSource:0}: Error finding container 12fee6f35b14bc15b5fafdccbebc36cf66bc3d255930b93931d451514b65cf00: Status 404 returned error can't find the container with id 12fee6f35b14bc15b5fafdccbebc36cf66bc3d255930b93931d451514b65cf00 Oct 30 19:25:37.247766 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:37.247733 4904 generic.go:356] "Generic (PLEG): container finished" podID="5ab94ee3-20b2-46e8-84a7-285824abb7ae" containerID="508fcc77464e5aea14fa7aef74390f118bfe68327d851fd4ad364eb28891d30d" exitCode=0 Oct 30 19:25:37.248191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:37.247817 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" event={"ID":"5ab94ee3-20b2-46e8-84a7-285824abb7ae","Type":"ContainerDied","Data":"508fcc77464e5aea14fa7aef74390f118bfe68327d851fd4ad364eb28891d30d"} Oct 30 19:25:37.249634 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:37.249615 4904 generic.go:356] "Generic (PLEG): container finished" podID="2b71c1f3-ad2b-4aef-8256-693b02b702af" containerID="8f0a50b765300a73da0e03392c5c2183be5a881295b45377aeeeff88d5a6ada7" exitCode=0 Oct 30 19:25:37.249727 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:37.249688 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" event={"ID":"2b71c1f3-ad2b-4aef-8256-693b02b702af","Type":"ContainerDied","Data":"8f0a50b765300a73da0e03392c5c2183be5a881295b45377aeeeff88d5a6ada7"} Oct 30 19:25:37.249727 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:37.249714 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" event={"ID":"2b71c1f3-ad2b-4aef-8256-693b02b702af","Type":"ContainerStarted","Data":"12fee6f35b14bc15b5fafdccbebc36cf66bc3d255930b93931d451514b65cf00"} Oct 30 19:25:38.263419 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:38.263386 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" event={"ID":"5ab94ee3-20b2-46e8-84a7-285824abb7ae","Type":"ContainerStarted","Data":"c73af9956aeda4887551c8b8b2cb85ee872b2cda1f143a52b7d31d821b5ff296"} Oct 30 19:25:38.263892 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:38.263476 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:38.265458 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:38.265434 4904 generic.go:356] "Generic (PLEG): container finished" podID="2b71c1f3-ad2b-4aef-8256-693b02b702af" containerID="d4240d913931db6b96af9e69694a246c0400f6ae171b835647a716f620ec6635" exitCode=0 Oct 30 19:25:38.265602 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:38.265494 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" event={"ID":"2b71c1f3-ad2b-4aef-8256-693b02b702af","Type":"ContainerDied","Data":"d4240d913931db6b96af9e69694a246c0400f6ae171b835647a716f620ec6635"} Oct 30 19:25:38.331655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:38.331611 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" podStartSLOduration=2.620024275 podStartE2EDuration="6.331593989s" totalImagesPullingTime="3.135788765s" totalInitContainerRuntime="575.780949ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:38.328034694 +0000 UTC m=+299.241639329" watchObservedRunningTime="2026-10-30 19:25:38.331593989 +0000 UTC m=+299.245198605" Oct 30 19:25:39.230059 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:39.230032 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:25:39.230645 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:39.230621 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:25:39.246446 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:39.246428 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:25:39.246811 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:39.246789 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:25:39.272432 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:39.272409 4904 generic.go:356] "Generic (PLEG): container finished" podID="2b71c1f3-ad2b-4aef-8256-693b02b702af" containerID="0106ba1a2b940e399f82992e3f59706e74cb220cc9b6b15139331cbdac5196cd" exitCode=0 Oct 30 19:25:39.272815 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:39.272462 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" event={"ID":"2b71c1f3-ad2b-4aef-8256-693b02b702af","Type":"ContainerDied","Data":"0106ba1a2b940e399f82992e3f59706e74cb220cc9b6b15139331cbdac5196cd"} Oct 30 19:25:40.419071 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:40.419042 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:40.503920 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:40.503896 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-util\" (UniqueName: \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-util\") pod \"2b71c1f3-ad2b-4aef-8256-693b02b702af\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " Oct 30 19:25:40.504026 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:40.503984 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b71c1f3-ad2b-4aef-8256-693b02b702af-kube-api-access-rl5n7\" (UniqueName: \"kubernetes.io/projected/2b71c1f3-ad2b-4aef-8256-693b02b702af-kube-api-access-rl5n7\") pod \"2b71c1f3-ad2b-4aef-8256-693b02b702af\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " Oct 30 19:25:40.504072 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:40.504061 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-bundle\") pod \"2b71c1f3-ad2b-4aef-8256-693b02b702af\" (UID: \"2b71c1f3-ad2b-4aef-8256-693b02b702af\") " Oct 30 19:25:40.504738 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:40.504704 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-bundle" pod "2b71c1f3-ad2b-4aef-8256-693b02b702af" (UID: "2b71c1f3-ad2b-4aef-8256-693b02b702af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:40.506120 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:40.506078 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b71c1f3-ad2b-4aef-8256-693b02b702af-kube-api-access-rl5n7" pod "2b71c1f3-ad2b-4aef-8256-693b02b702af" (UID: "2b71c1f3-ad2b-4aef-8256-693b02b702af"). InnerVolumeSpecName "kube-api-access-rl5n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:40.510747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:40.510712 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-util" pod "2b71c1f3-ad2b-4aef-8256-693b02b702af" (UID: "2b71c1f3-ad2b-4aef-8256-693b02b702af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:40.605559 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:40.605498 4904 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-bundle\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:40.605559 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:40.605532 4904 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b71c1f3-ad2b-4aef-8256-693b02b702af-util\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:40.605559 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:40.605541 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5n7\" (UniqueName: \"kubernetes.io/projected/2b71c1f3-ad2b-4aef-8256-693b02b702af-kube-api-access-rl5n7\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:25:41.139451 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.139423 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz"] Oct 30 19:25:41.139769 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.139756 4904 cpu_manager.go:414] "containerMap: removing container" podUID="2b71c1f3-ad2b-4aef-8256-693b02b702af" containerName="pull" Oct 30 19:25:41.139813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.139771 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b71c1f3-ad2b-4aef-8256-693b02b702af" containerName="pull" Oct 30 19:25:41.139813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.139782 4904 cpu_manager.go:414] "containerMap: removing container" podUID="2b71c1f3-ad2b-4aef-8256-693b02b702af" containerName="extract" Oct 30 19:25:41.139813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.139788 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b71c1f3-ad2b-4aef-8256-693b02b702af" containerName="extract" Oct 30 19:25:41.139908 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.139817 4904 cpu_manager.go:414] "containerMap: removing container" podUID="2b71c1f3-ad2b-4aef-8256-693b02b702af" containerName="util" Oct 30 19:25:41.139908 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.139823 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b71c1f3-ad2b-4aef-8256-693b02b702af" containerName="util" Oct 30 19:25:41.139908 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.139877 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b71c1f3-ad2b-4aef-8256-693b02b702af" containerName="extract" Oct 30 19:25:41.143212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.143196 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" Oct 30 19:25:41.155091 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.155069 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz"] Oct 30 19:25:41.168235 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.168216 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 19:25:41.168335 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.168255 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:25:41.168335 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.168299 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-42xfd\"" Oct 30 19:25:41.209251 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.209230 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0ca9a9e-617d-4e53-b500-7758f40f5e8a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz\" (UID: \"d0ca9a9e-617d-4e53-b500-7758f40f5e8a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" Oct 30 19:25:41.209336 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.209265 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nqgb\" (UniqueName: \"kubernetes.io/projected/d0ca9a9e-617d-4e53-b500-7758f40f5e8a-kube-api-access-6nqgb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz\" (UID: \"d0ca9a9e-617d-4e53-b500-7758f40f5e8a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" Oct 30 19:25:41.284592 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.284569 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" event={"ID":"2b71c1f3-ad2b-4aef-8256-693b02b702af","Type":"ContainerDied","Data":"12fee6f35b14bc15b5fafdccbebc36cf66bc3d255930b93931d451514b65cf00"} Oct 30 19:25:41.284669 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.284603 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12fee6f35b14bc15b5fafdccbebc36cf66bc3d255930b93931d451514b65cf00" Oct 30 19:25:41.284669 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.284579 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtmhq" Oct 30 19:25:41.310214 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.310191 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nqgb\" (UniqueName: \"kubernetes.io/projected/d0ca9a9e-617d-4e53-b500-7758f40f5e8a-kube-api-access-6nqgb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz\" (UID: \"d0ca9a9e-617d-4e53-b500-7758f40f5e8a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" Oct 30 19:25:41.310453 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.310423 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0ca9a9e-617d-4e53-b500-7758f40f5e8a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz\" (UID: \"d0ca9a9e-617d-4e53-b500-7758f40f5e8a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" Oct 30 19:25:41.310720 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.310703 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0ca9a9e-617d-4e53-b500-7758f40f5e8a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz\" (UID: \"d0ca9a9e-617d-4e53-b500-7758f40f5e8a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" Oct 30 19:25:41.334361 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.334343 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nqgb\" (UniqueName: \"kubernetes.io/projected/d0ca9a9e-617d-4e53-b500-7758f40f5e8a-kube-api-access-6nqgb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz\" (UID: \"d0ca9a9e-617d-4e53-b500-7758f40f5e8a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" Oct 30 19:25:41.456985 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.456966 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" Oct 30 19:25:41.792060 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:41.792040 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz"] Oct 30 19:25:41.794242 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:25:41.794204 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ca9a9e_617d_4e53_b500_7758f40f5e8a.slice/crio-80700bbefc50de3d6c61b9e56d58cffa51791f443803bdd7bcdf64c9fefa10e1 WatchSource:0}: Error finding container 80700bbefc50de3d6c61b9e56d58cffa51791f443803bdd7bcdf64c9fefa10e1: Status 404 returned error can't find the container with id 80700bbefc50de3d6c61b9e56d58cffa51791f443803bdd7bcdf64c9fefa10e1 Oct 30 19:25:42.291864 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:42.291835 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" event={"ID":"d0ca9a9e-617d-4e53-b500-7758f40f5e8a","Type":"ContainerStarted","Data":"80700bbefc50de3d6c61b9e56d58cffa51791f443803bdd7bcdf64c9fefa10e1"} Oct 30 19:25:44.304930 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:44.304855 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" event={"ID":"d0ca9a9e-617d-4e53-b500-7758f40f5e8a","Type":"ContainerStarted","Data":"c7365865690d69e46dbd5e6baffde4948b5fdcb899563512721517756dc8bc3a"} Oct 30 19:25:44.377278 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:44.377226 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vp6xz" podStartSLOduration=1.211300988 podStartE2EDuration="3.377211345s" totalImagesPullingTime="2.165910357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:44.362357972 +0000 UTC m=+305.275962591" watchObservedRunningTime="2026-10-30 19:25:44.377211345 +0000 UTC m=+305.290815940" Oct 30 19:25:49.274789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:49.274714 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-ngmkv" Oct 30 19:25:53.642841 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.642807 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f"] Oct 30 19:25:53.646377 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.646361 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f" Oct 30 19:25:53.679725 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.679700 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 19:25:53.679725 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.679704 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:25:53.679894 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.679743 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:25:53.679894 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.679784 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zj725\"" Oct 30 19:25:53.681509 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.681492 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f"] Oct 30 19:25:53.739270 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.739248 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-79x29"] Oct 30 19:25:53.742654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.742639 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" Oct 30 19:25:53.764055 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.764035 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-79x29"] Oct 30 19:25:53.770005 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.769985 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 19:25:53.770097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.770008 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 19:25:53.770097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.770051 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-wf8pr\"" Oct 30 19:25:53.804082 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.804060 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2568922e-f36d-41c4-80b5-362fc955a402-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-79x29\" (UID: \"2568922e-f36d-41c4-80b5-362fc955a402\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" Oct 30 19:25:53.804188 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.804092 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdwcg\" (UniqueName: \"kubernetes.io/projected/2568922e-f36d-41c4-80b5-362fc955a402-kube-api-access-sdwcg\") pod \"cert-manager-webhook-5dbbbf75cc-79x29\" (UID: \"2568922e-f36d-41c4-80b5-362fc955a402\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" Oct 30 19:25:53.804231 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.804192 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77r48\" (UniqueName: \"kubernetes.io/projected/bcbc334e-5d81-4772-abc0-4aa196f064b6-kube-api-access-77r48\") pod \"dns-operator-controller-manager-8557cbdb84-t8d5f\" (UID: \"bcbc334e-5d81-4772-abc0-4aa196f064b6\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f" Oct 30 19:25:53.904670 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.904623 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2568922e-f36d-41c4-80b5-362fc955a402-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-79x29\" (UID: \"2568922e-f36d-41c4-80b5-362fc955a402\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" Oct 30 19:25:53.904670 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.904647 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdwcg\" (UniqueName: \"kubernetes.io/projected/2568922e-f36d-41c4-80b5-362fc955a402-kube-api-access-sdwcg\") pod \"cert-manager-webhook-5dbbbf75cc-79x29\" (UID: \"2568922e-f36d-41c4-80b5-362fc955a402\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" Oct 30 19:25:53.904817 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.904672 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77r48\" (UniqueName: \"kubernetes.io/projected/bcbc334e-5d81-4772-abc0-4aa196f064b6-kube-api-access-77r48\") pod \"dns-operator-controller-manager-8557cbdb84-t8d5f\" (UID: \"bcbc334e-5d81-4772-abc0-4aa196f064b6\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f" Oct 30 19:25:53.928133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.928095 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2568922e-f36d-41c4-80b5-362fc955a402-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-79x29\" (UID: \"2568922e-f36d-41c4-80b5-362fc955a402\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" Oct 30 19:25:53.928608 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.928586 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77r48\" (UniqueName: \"kubernetes.io/projected/bcbc334e-5d81-4772-abc0-4aa196f064b6-kube-api-access-77r48\") pod \"dns-operator-controller-manager-8557cbdb84-t8d5f\" (UID: \"bcbc334e-5d81-4772-abc0-4aa196f064b6\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f" Oct 30 19:25:53.929377 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.929358 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdwcg\" (UniqueName: \"kubernetes.io/projected/2568922e-f36d-41c4-80b5-362fc955a402-kube-api-access-sdwcg\") pod \"cert-manager-webhook-5dbbbf75cc-79x29\" (UID: \"2568922e-f36d-41c4-80b5-362fc955a402\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" Oct 30 19:25:53.959263 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:53.959244 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f" Oct 30 19:25:54.066825 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:54.066791 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" Oct 30 19:25:54.287677 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:25:54.287639 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbc334e_5d81_4772_abc0_4aa196f064b6.slice/crio-c0ea3c632c40ca75679b009e92f17d0c9ae7bb03795343d989908713c6ffd9a4 WatchSource:0}: Error finding container c0ea3c632c40ca75679b009e92f17d0c9ae7bb03795343d989908713c6ffd9a4: Status 404 returned error can't find the container with id c0ea3c632c40ca75679b009e92f17d0c9ae7bb03795343d989908713c6ffd9a4 Oct 30 19:25:54.290360 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:54.290344 4904 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:25:54.291033 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:54.291009 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f"] Oct 30 19:25:54.354339 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:54.354307 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f" event={"ID":"bcbc334e-5d81-4772-abc0-4aa196f064b6","Type":"ContainerStarted","Data":"c0ea3c632c40ca75679b009e92f17d0c9ae7bb03795343d989908713c6ffd9a4"} Oct 30 19:25:54.378652 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:54.378635 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-79x29"] Oct 30 19:25:54.380593 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:25:54.380564 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2568922e_f36d_41c4_80b5_362fc955a402.slice/crio-99627e9a6188e9be3cf5042a1b69c91edad5cbf3868b2eb8707839ab04ff772e WatchSource:0}: Error finding container 99627e9a6188e9be3cf5042a1b69c91edad5cbf3868b2eb8707839ab04ff772e: Status 404 returned error can't find the container with id 99627e9a6188e9be3cf5042a1b69c91edad5cbf3868b2eb8707839ab04ff772e Oct 30 19:25:54.507405 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:25:54.507381 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:25:55.363481 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:55.363442 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" event={"ID":"2568922e-f36d-41c4-80b5-362fc955a402","Type":"ContainerStarted","Data":"99627e9a6188e9be3cf5042a1b69c91edad5cbf3868b2eb8707839ab04ff772e"} Oct 30 19:25:56.889727 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:56.889705 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b"] Oct 30 19:25:56.894905 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:56.894881 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b" Oct 30 19:25:56.922945 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:56.922923 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b"] Oct 30 19:25:56.924694 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:56.924662 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-b97v9\"" Oct 30 19:25:57.035334 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.035296 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5sm\" (UniqueName: \"kubernetes.io/projected/b85b9f89-71c2-4664-aaad-29a15ef6f6b2-kube-api-access-gg5sm\") pod \"limitador-operator-controller-manager-5778bb4449-lln9b\" (UID: \"b85b9f89-71c2-4664-aaad-29a15ef6f6b2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b" Oct 30 19:25:57.136148 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.136062 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5sm\" (UniqueName: \"kubernetes.io/projected/b85b9f89-71c2-4664-aaad-29a15ef6f6b2-kube-api-access-gg5sm\") pod \"limitador-operator-controller-manager-5778bb4449-lln9b\" (UID: \"b85b9f89-71c2-4664-aaad-29a15ef6f6b2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b" Oct 30 19:25:57.172347 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.172322 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5sm\" (UniqueName: \"kubernetes.io/projected/b85b9f89-71c2-4664-aaad-29a15ef6f6b2-kube-api-access-gg5sm\") pod \"limitador-operator-controller-manager-5778bb4449-lln9b\" (UID: \"b85b9f89-71c2-4664-aaad-29a15ef6f6b2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b" Oct 30 19:25:57.235904 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.235880 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b" Oct 30 19:25:57.376788 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.376747 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f" event={"ID":"bcbc334e-5d81-4772-abc0-4aa196f064b6","Type":"ContainerStarted","Data":"52157b1bf5fb203049bd781cac2146563bb8621c1f463418b568217fc47964be"} Oct 30 19:25:57.376947 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.376806 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f" Oct 30 19:25:57.378332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.378308 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" event={"ID":"2568922e-f36d-41c4-80b5-362fc955a402","Type":"ContainerStarted","Data":"2c8d375beb1b026e460236d9695b747b4132f361016021e3adc01b4235158673"} Oct 30 19:25:57.378431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.378343 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" Oct 30 19:25:57.428402 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.428361 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f" podStartSLOduration=1.8361227690000002 podStartE2EDuration="4.428349139s" totalImagesPullingTime="2.59222637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:57.42764084 +0000 UTC m=+318.341245456" watchObservedRunningTime="2026-10-30 19:25:57.428349139 +0000 UTC m=+318.341953755" Oct 30 19:25:57.470650 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.470609 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" podStartSLOduration=1.8031325759999999 podStartE2EDuration="4.470594824s" totalImagesPullingTime="2.667462248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:57.470174572 +0000 UTC m=+318.383779191" watchObservedRunningTime="2026-10-30 19:25:57.470594824 +0000 UTC m=+318.384199438" Oct 30 19:25:57.542783 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:57.542764 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b"] Oct 30 19:25:57.544701 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:25:57.544682 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85b9f89_71c2_4664_aaad_29a15ef6f6b2.slice/crio-ac55314289d19cd9b4d6f2a189598b181807f58c0b815cf74cede54b2a87cf3e WatchSource:0}: Error finding container ac55314289d19cd9b4d6f2a189598b181807f58c0b815cf74cede54b2a87cf3e: Status 404 returned error can't find the container with id ac55314289d19cd9b4d6f2a189598b181807f58c0b815cf74cede54b2a87cf3e Oct 30 19:25:58.387505 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:58.387459 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b" event={"ID":"b85b9f89-71c2-4664-aaad-29a15ef6f6b2","Type":"ContainerStarted","Data":"ac55314289d19cd9b4d6f2a189598b181807f58c0b815cf74cede54b2a87cf3e"} Oct 30 19:25:59.393950 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:59.393856 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b" event={"ID":"b85b9f89-71c2-4664-aaad-29a15ef6f6b2","Type":"ContainerStarted","Data":"43f4060fb7fcebeea2e6f4e56b9e343cd44eb67bb208280436d849ec7516d34e"} Oct 30 19:25:59.394417 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:59.394009 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b" Oct 30 19:25:59.441582 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:25:59.439363 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b" podStartSLOduration=2.021453151 podStartE2EDuration="3.439344554s" totalImagesPullingTime="1.417891403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:59.436862896 +0000 UTC m=+320.350467512" watchObservedRunningTime="2026-10-30 19:25:59.439344554 +0000 UTC m=+320.352949173" Oct 30 19:26:03.319859 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.319819 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn"] Oct 30 19:26:03.323708 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.323689 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:26:03.340013 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.339985 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn"] Oct 30 19:26:03.350689 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.350670 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zh7c7\"" Oct 30 19:26:03.389984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.389967 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" Oct 30 19:26:03.483093 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.483062 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vtgpn\" (UID: \"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:26:03.483314 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.483295 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vpt\" (UniqueName: \"kubernetes.io/projected/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-kube-api-access-t4vpt\") pod \"kuadrant-operator-controller-manager-678d87766f-vtgpn\" (UID: \"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:26:03.584059 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.583991 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vtgpn\" (UID: \"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:26:03.584059 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.584054 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vpt\" (UniqueName: \"kubernetes.io/projected/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-kube-api-access-t4vpt\") pod \"kuadrant-operator-controller-manager-678d87766f-vtgpn\" (UID: \"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:26:03.584383 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.584363 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vtgpn\" (UID: \"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:26:03.604622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.604598 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vpt\" (UniqueName: \"kubernetes.io/projected/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-kube-api-access-t4vpt\") pod \"kuadrant-operator-controller-manager-678d87766f-vtgpn\" (UID: \"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:26:03.637582 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.637558 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:26:03.965916 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:03.965886 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn"] Oct 30 19:26:03.980761 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:26:03.980736 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ad1f5f_8f45_463b_9c5e_b0f9f06315c4.slice/crio-2ed8c7d981ffc9de4dc2ae2a8d83b612fcdd2f5e5b96013b3ef6d73081138365 WatchSource:0}: Error finding container 2ed8c7d981ffc9de4dc2ae2a8d83b612fcdd2f5e5b96013b3ef6d73081138365: Status 404 returned error can't find the container with id 2ed8c7d981ffc9de4dc2ae2a8d83b612fcdd2f5e5b96013b3ef6d73081138365 Oct 30 19:26:04.421041 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:04.420994 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" event={"ID":"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4","Type":"ContainerStarted","Data":"2ed8c7d981ffc9de4dc2ae2a8d83b612fcdd2f5e5b96013b3ef6d73081138365"} Oct 30 19:26:05.928813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:05.928780 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-whljk"] Oct 30 19:26:05.932636 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:05.932615 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-whljk" Oct 30 19:26:05.965884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:05.965859 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-whljk"] Oct 30 19:26:06.019202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:06.019171 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6j7h9\"" Oct 30 19:26:06.104181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:06.104147 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjc5n\" (UniqueName: \"kubernetes.io/projected/84ea9711-daf8-4c43-81c7-8a31cc88d45f-kube-api-access-kjc5n\") pod \"authorino-operator-7b7bb4bc77-whljk\" (UID: \"84ea9711-daf8-4c43-81c7-8a31cc88d45f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-whljk" Oct 30 19:26:06.205435 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:06.205349 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjc5n\" (UniqueName: \"kubernetes.io/projected/84ea9711-daf8-4c43-81c7-8a31cc88d45f-kube-api-access-kjc5n\") pod \"authorino-operator-7b7bb4bc77-whljk\" (UID: \"84ea9711-daf8-4c43-81c7-8a31cc88d45f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-whljk" Oct 30 19:26:06.231870 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:06.231842 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjc5n\" (UniqueName: \"kubernetes.io/projected/84ea9711-daf8-4c43-81c7-8a31cc88d45f-kube-api-access-kjc5n\") pod \"authorino-operator-7b7bb4bc77-whljk\" (UID: \"84ea9711-daf8-4c43-81c7-8a31cc88d45f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-whljk" Oct 30 19:26:06.248531 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:06.248504 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-whljk" Oct 30 19:26:06.560600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:06.560521 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-whljk"] Oct 30 19:26:06.566368 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:26:06.566330 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ea9711_daf8_4c43_81c7_8a31cc88d45f.slice/crio-023a0c0cd0c8620b358a47f3164247705d640fff0226a0f44b5e88e6fd7fb16d WatchSource:0}: Error finding container 023a0c0cd0c8620b358a47f3164247705d640fff0226a0f44b5e88e6fd7fb16d: Status 404 returned error can't find the container with id 023a0c0cd0c8620b358a47f3164247705d640fff0226a0f44b5e88e6fd7fb16d Oct 30 19:26:07.441605 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:07.441564 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-whljk" event={"ID":"84ea9711-daf8-4c43-81c7-8a31cc88d45f","Type":"ContainerStarted","Data":"023a0c0cd0c8620b358a47f3164247705d640fff0226a0f44b5e88e6fd7fb16d"} Oct 30 19:26:08.389871 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.389843 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t8d5f" Oct 30 19:26:08.449111 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.449070 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-whljk" event={"ID":"84ea9711-daf8-4c43-81c7-8a31cc88d45f","Type":"ContainerStarted","Data":"fefaf5918eaf8a613258a46b4b0e17d08fe7e0edd15d7c163c05a34e3a2f8e79"} Oct 30 19:26:08.449655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.449191 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-whljk" Oct 30 19:26:08.451393 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.451364 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" event={"ID":"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4","Type":"ContainerStarted","Data":"9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f"} Oct 30 19:26:08.451551 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.451463 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:26:08.499921 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.499858 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-whljk" podStartSLOduration=2.419888689 podStartE2EDuration="3.49984343s" totalImagesPullingTime="1.079954741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:08.496439903 +0000 UTC m=+329.410044520" watchObservedRunningTime="2026-10-30 19:26:08.49984343 +0000 UTC m=+329.413448047" Oct 30 19:26:08.546174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.546097 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" podStartSLOduration=2.058680476 podStartE2EDuration="5.546079643s" totalImagesPullingTime="3.487399167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:08.544952929 +0000 UTC m=+329.458557546" watchObservedRunningTime="2026-10-30 19:26:08.546079643 +0000 UTC m=+329.459684263" Oct 30 19:26:08.894688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.894646 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt"] Oct 30 19:26:08.898350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.898329 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:08.920299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.920267 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mwfzb\"" Oct 30 19:26:08.923013 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:08.922993 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt"] Oct 30 19:26:09.032002 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.031976 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.032134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.032019 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.032134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.032046 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.032134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.032078 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbz59\" (UniqueName: \"kubernetes.io/projected/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-kube-api-access-vbz59\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.032318 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.032184 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.032318 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.032276 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.032436 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.032314 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.032436 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.032346 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.032436 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.032385 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.133750 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.133722 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.133893 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.133756 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.133893 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.133774 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.133893 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.133802 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.133893 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.133863 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.133893 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.133888 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.134174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.133911 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.134174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.133942 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbz59\" (UniqueName: \"kubernetes.io/projected/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-kube-api-access-vbz59\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.134174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.133974 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.134326 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.134175 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.134480 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.134450 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.134606 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.134507 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.134733 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.134709 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.134825 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.134749 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.136541 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.136513 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.136652 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.136632 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.155883 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.155822 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbz59\" (UniqueName: \"kubernetes.io/projected/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-kube-api-access-vbz59\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.155989 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.155883 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e980f2ff-8bfb-4fbf-9bcc-d72274125bc6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fqv2lt\" (UID: \"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.211128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.211083 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:09.535934 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.535909 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt"] Oct 30 19:26:09.537805 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:26:09.537780 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode980f2ff_8bfb_4fbf_9bcc_d72274125bc6.slice/crio-3e830cf201d035a6ee7215aa4926db2f2158a89a037b1d23a72e4d458068d009 WatchSource:0}: Error finding container 3e830cf201d035a6ee7215aa4926db2f2158a89a037b1d23a72e4d458068d009: Status 404 returned error can't find the container with id 3e830cf201d035a6ee7215aa4926db2f2158a89a037b1d23a72e4d458068d009 Oct 30 19:26:09.539924 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.539892 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:26:09.540027 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.539980 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:26:09.540092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:09.540034 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:26:10.400876 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:10.400848 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-lln9b" Oct 30 19:26:10.463922 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:10.463887 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" event={"ID":"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6","Type":"ContainerStarted","Data":"c5b2b51b9f107894f70fac87137382df0e23f280011e20d24866f52c7272c6c9"} Oct 30 19:26:10.464075 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:10.463936 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" event={"ID":"e980f2ff-8bfb-4fbf-9bcc-d72274125bc6","Type":"ContainerStarted","Data":"3e830cf201d035a6ee7215aa4926db2f2158a89a037b1d23a72e4d458068d009"} Oct 30 19:26:10.514446 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:10.514388 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" podStartSLOduration=2.514371042 podStartE2EDuration="2.514371042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:10.512399914 +0000 UTC m=+331.426004548" watchObservedRunningTime="2026-10-30 19:26:10.514371042 +0000 UTC m=+331.427975661" Oct 30 19:26:11.193000 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.192966 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h4fdc"] Oct 30 19:26:11.195438 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.195415 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" Oct 30 19:26:11.212016 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.211994 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:11.221211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.221191 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h4fdc"] Oct 30 19:26:11.221310 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.221262 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:11.225871 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.225855 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dvjb7\"" Oct 30 19:26:11.355504 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.355475 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqhf5\" (UniqueName: \"kubernetes.io/projected/999de8b6-2491-49dd-9008-b659915aeca4-kube-api-access-vqhf5\") pod \"authorino-56c6c9ddd5-h4fdc\" (UID: \"999de8b6-2491-49dd-9008-b659915aeca4\") " pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" Oct 30 19:26:11.456140 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.456033 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqhf5\" (UniqueName: \"kubernetes.io/projected/999de8b6-2491-49dd-9008-b659915aeca4-kube-api-access-vqhf5\") pod \"authorino-56c6c9ddd5-h4fdc\" (UID: \"999de8b6-2491-49dd-9008-b659915aeca4\") " pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" Oct 30 19:26:11.468933 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.468903 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:11.469838 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.469823 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fqv2lt" Oct 30 19:26:11.478360 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.478331 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqhf5\" (UniqueName: \"kubernetes.io/projected/999de8b6-2491-49dd-9008-b659915aeca4-kube-api-access-vqhf5\") pod \"authorino-56c6c9ddd5-h4fdc\" (UID: \"999de8b6-2491-49dd-9008-b659915aeca4\") " pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" Oct 30 19:26:11.507364 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.507342 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" Oct 30 19:26:11.808381 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:11.808360 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h4fdc"] Oct 30 19:26:11.809989 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:26:11.809957 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod999de8b6_2491_49dd_9008_b659915aeca4.slice/crio-c844af3fe9f640953e9dd55599bc2a4c2a524533cb7c6549343d736add08e697 WatchSource:0}: Error finding container c844af3fe9f640953e9dd55599bc2a4c2a524533cb7c6549343d736add08e697: Status 404 returned error can't find the container with id c844af3fe9f640953e9dd55599bc2a4c2a524533cb7c6549343d736add08e697 Oct 30 19:26:12.476130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:12.476065 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" event={"ID":"999de8b6-2491-49dd-9008-b659915aeca4","Type":"ContainerStarted","Data":"c844af3fe9f640953e9dd55599bc2a4c2a524533cb7c6549343d736add08e697"} Oct 30 19:26:13.078370 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.078337 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss"] Oct 30 19:26:13.082427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.082405 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.101831 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.101806 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss"] Oct 30 19:26:13.112286 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.112254 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:26:13.112286 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.112274 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:26:13.139573 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.139543 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 19:26:13.139690 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.139620 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-tcgf6\"" Oct 30 19:26:13.171506 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.171476 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/355589dc-40b8-4ad2-8566-9f7eb2ce069f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.171619 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.171526 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6gh\" (UniqueName: \"kubernetes.io/projected/355589dc-40b8-4ad2-8566-9f7eb2ce069f-kube-api-access-ws6gh\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.171685 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.171657 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/355589dc-40b8-4ad2-8566-9f7eb2ce069f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.171747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.171693 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/355589dc-40b8-4ad2-8566-9f7eb2ce069f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.272280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.272244 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/355589dc-40b8-4ad2-8566-9f7eb2ce069f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.272460 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.272340 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/355589dc-40b8-4ad2-8566-9f7eb2ce069f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.272460 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.272374 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6gh\" (UniqueName: \"kubernetes.io/projected/355589dc-40b8-4ad2-8566-9f7eb2ce069f-kube-api-access-ws6gh\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.272460 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.272415 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/355589dc-40b8-4ad2-8566-9f7eb2ce069f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.272686 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.272666 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/355589dc-40b8-4ad2-8566-9f7eb2ce069f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.272747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.272704 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/355589dc-40b8-4ad2-8566-9f7eb2ce069f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.273267 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.273247 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/355589dc-40b8-4ad2-8566-9f7eb2ce069f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.295465 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.295444 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6gh\" (UniqueName: \"kubernetes.io/projected/355589dc-40b8-4ad2-8566-9f7eb2ce069f-kube-api-access-ws6gh\") pod \"kserve-module-controller-manager-7566cf5fc8-fl9ss\" (UID: \"355589dc-40b8-4ad2-8566-9f7eb2ce069f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.399482 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.399434 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:13.484722 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.484689 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" event={"ID":"999de8b6-2491-49dd-9008-b659915aeca4","Type":"ContainerStarted","Data":"0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2"} Oct 30 19:26:13.526002 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.525953 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" podStartSLOduration=0.97024 podStartE2EDuration="2.525940737s" totalImagesPullingTime="1.555700737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:13.525183889 +0000 UTC m=+334.438788506" watchObservedRunningTime="2026-10-30 19:26:13.525940737 +0000 UTC m=+334.439545354" Oct 30 19:26:13.716665 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:13.716643 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss"] Oct 30 19:26:13.718358 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:26:13.718332 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355589dc_40b8_4ad2_8566_9f7eb2ce069f.slice/crio-c91881fed087435b1ad527cd161bb7f6873136a12b5e7e86a7bb3abdc548f828 WatchSource:0}: Error finding container c91881fed087435b1ad527cd161bb7f6873136a12b5e7e86a7bb3abdc548f828: Status 404 returned error can't find the container with id c91881fed087435b1ad527cd161bb7f6873136a12b5e7e86a7bb3abdc548f828 Oct 30 19:26:14.495139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:14.495064 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" event={"ID":"355589dc-40b8-4ad2-8566-9f7eb2ce069f","Type":"ContainerStarted","Data":"c91881fed087435b1ad527cd161bb7f6873136a12b5e7e86a7bb3abdc548f828"} Oct 30 19:26:16.511800 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:16.511759 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" event={"ID":"355589dc-40b8-4ad2-8566-9f7eb2ce069f","Type":"ContainerStarted","Data":"ebb8b31f7b77afa955fbceeff2d5df933392bc92c5b32905cfbc364d133ea188"} Oct 30 19:26:16.512321 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:16.511913 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:16.569761 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:16.569708 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" podStartSLOduration=1.237747214 podStartE2EDuration="3.569695282s" totalImagesPullingTime="2.331948068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:16.568678415 +0000 UTC m=+337.482283033" watchObservedRunningTime="2026-10-30 19:26:16.569695282 +0000 UTC m=+337.483299898" Oct 30 19:26:19.458667 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:19.458640 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-whljk" Oct 30 19:26:19.459077 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:19.458861 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:26:24.509879 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:26:24.509841 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:26:37.518408 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:26:37.518378 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fl9ss" Oct 30 19:26:54.508722 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:26:54.508684 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:27:09.989144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:09.989096 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9kdnq"] Oct 30 19:27:09.995966 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:09.995944 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" Oct 30 19:27:10.010338 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.010315 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9kdnq"] Oct 30 19:27:10.021193 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.021175 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-46jz6\"" Oct 30 19:27:10.024808 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.024790 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 19:27:10.032977 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.032954 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t"] Oct 30 19:27:10.036775 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.036759 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:27:10.055988 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.055963 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t"] Oct 30 19:27:10.070945 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.070921 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 19:27:10.071038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.070946 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2hw68\"" Oct 30 19:27:10.078624 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.078604 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-prj9w"] Oct 30 19:27:10.082251 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.082233 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" Oct 30 19:27:10.103287 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.103246 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-prj9w"] Oct 30 19:27:10.120305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.120283 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-795kr"] Oct 30 19:27:10.123673 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.123657 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.125875 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.125856 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0dc02fe-b7e5-419e-80af-d0c9df5b6742-cert\") pod \"kserve-controller-manager-776cc66d47-9kdnq\" (UID: \"b0dc02fe-b7e5-419e-80af-d0c9df5b6742\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" Oct 30 19:27:10.125977 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.125903 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnst\" (UniqueName: \"kubernetes.io/projected/a72bdca4-b46b-4a16-b57a-7bbf06de541c-kube-api-access-ddnst\") pod \"llmisvc-controller-manager-869d5dcc5d-zgf8t\" (UID: \"a72bdca4-b46b-4a16-b57a-7bbf06de541c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:27:10.126044 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.126020 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a72bdca4-b46b-4a16-b57a-7bbf06de541c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zgf8t\" (UID: \"a72bdca4-b46b-4a16-b57a-7bbf06de541c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:27:10.126127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.126085 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cql7h\" (UniqueName: \"kubernetes.io/projected/b0dc02fe-b7e5-419e-80af-d0c9df5b6742-kube-api-access-cql7h\") pod \"kserve-controller-manager-776cc66d47-9kdnq\" (UID: \"b0dc02fe-b7e5-419e-80af-d0c9df5b6742\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" Oct 30 19:27:10.147254 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.147229 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-795kr"] Oct 30 19:27:10.170314 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.170290 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 19:27:10.170458 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.170325 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 19:27:10.170458 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.170410 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-s6f4w\"" Oct 30 19:27:10.170458 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.170453 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6rpbv\"" Oct 30 19:27:10.170577 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.170463 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 19:27:10.226564 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.226541 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnst\" (UniqueName: \"kubernetes.io/projected/a72bdca4-b46b-4a16-b57a-7bbf06de541c-kube-api-access-ddnst\") pod \"llmisvc-controller-manager-869d5dcc5d-zgf8t\" (UID: \"a72bdca4-b46b-4a16-b57a-7bbf06de541c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:27:10.226665 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.226579 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad4552e-0da1-49db-80b8-fb4573db1d60-metrics-certs\") pod \"odh-model-controller-769875db56-795kr\" (UID: \"1ad4552e-0da1-49db-80b8-fb4573db1d60\") " pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.226665 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.226602 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a72bdca4-b46b-4a16-b57a-7bbf06de541c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zgf8t\" (UID: \"a72bdca4-b46b-4a16-b57a-7bbf06de541c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:27:10.226665 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.226624 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ad4552e-0da1-49db-80b8-fb4573db1d60-cert\") pod \"odh-model-controller-769875db56-795kr\" (UID: \"1ad4552e-0da1-49db-80b8-fb4573db1d60\") " pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.226786 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.226696 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5nt\" (UniqueName: \"kubernetes.io/projected/c5898fda-c3ce-4907-a685-058d32307825-kube-api-access-cs5nt\") pod \"model-serving-api-75bd89fdfc-prj9w\" (UID: \"c5898fda-c3ce-4907-a685-058d32307825\") " pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" Oct 30 19:27:10.226786 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.226730 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cql7h\" (UniqueName: \"kubernetes.io/projected/b0dc02fe-b7e5-419e-80af-d0c9df5b6742-kube-api-access-cql7h\") pod \"kserve-controller-manager-776cc66d47-9kdnq\" (UID: \"b0dc02fe-b7e5-419e-80af-d0c9df5b6742\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" Oct 30 19:27:10.226786 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.226767 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bm5p\" (UniqueName: \"kubernetes.io/projected/1ad4552e-0da1-49db-80b8-fb4573db1d60-kube-api-access-5bm5p\") pod \"odh-model-controller-769875db56-795kr\" (UID: \"1ad4552e-0da1-49db-80b8-fb4573db1d60\") " pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.226902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.226797 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5898fda-c3ce-4907-a685-058d32307825-tls-certs\") pod \"model-serving-api-75bd89fdfc-prj9w\" (UID: \"c5898fda-c3ce-4907-a685-058d32307825\") " pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" Oct 30 19:27:10.226939 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.226907 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0dc02fe-b7e5-419e-80af-d0c9df5b6742-cert\") pod \"kserve-controller-manager-776cc66d47-9kdnq\" (UID: \"b0dc02fe-b7e5-419e-80af-d0c9df5b6742\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" Oct 30 19:27:10.229304 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.229280 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a72bdca4-b46b-4a16-b57a-7bbf06de541c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zgf8t\" (UID: \"a72bdca4-b46b-4a16-b57a-7bbf06de541c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:27:10.229398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.229323 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0dc02fe-b7e5-419e-80af-d0c9df5b6742-cert\") pod \"kserve-controller-manager-776cc66d47-9kdnq\" (UID: \"b0dc02fe-b7e5-419e-80af-d0c9df5b6742\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" Oct 30 19:27:10.247305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.247248 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnst\" (UniqueName: \"kubernetes.io/projected/a72bdca4-b46b-4a16-b57a-7bbf06de541c-kube-api-access-ddnst\") pod \"llmisvc-controller-manager-869d5dcc5d-zgf8t\" (UID: \"a72bdca4-b46b-4a16-b57a-7bbf06de541c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:27:10.247305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.247271 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cql7h\" (UniqueName: \"kubernetes.io/projected/b0dc02fe-b7e5-419e-80af-d0c9df5b6742-kube-api-access-cql7h\") pod \"kserve-controller-manager-776cc66d47-9kdnq\" (UID: \"b0dc02fe-b7e5-419e-80af-d0c9df5b6742\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" Oct 30 19:27:10.309238 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.309220 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" Oct 30 19:27:10.327580 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.327561 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bm5p\" (UniqueName: \"kubernetes.io/projected/1ad4552e-0da1-49db-80b8-fb4573db1d60-kube-api-access-5bm5p\") pod \"odh-model-controller-769875db56-795kr\" (UID: \"1ad4552e-0da1-49db-80b8-fb4573db1d60\") " pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.327710 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.327590 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5898fda-c3ce-4907-a685-058d32307825-tls-certs\") pod \"model-serving-api-75bd89fdfc-prj9w\" (UID: \"c5898fda-c3ce-4907-a685-058d32307825\") " pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" Oct 30 19:27:10.327710 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.327632 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad4552e-0da1-49db-80b8-fb4573db1d60-metrics-certs\") pod \"odh-model-controller-769875db56-795kr\" (UID: \"1ad4552e-0da1-49db-80b8-fb4573db1d60\") " pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.327835 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.327817 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ad4552e-0da1-49db-80b8-fb4573db1d60-cert\") pod \"odh-model-controller-769875db56-795kr\" (UID: \"1ad4552e-0da1-49db-80b8-fb4573db1d60\") " pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.327894 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.327864 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5nt\" (UniqueName: \"kubernetes.io/projected/c5898fda-c3ce-4907-a685-058d32307825-kube-api-access-cs5nt\") pod \"model-serving-api-75bd89fdfc-prj9w\" (UID: \"c5898fda-c3ce-4907-a685-058d32307825\") " pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" Oct 30 19:27:10.330040 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.330023 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5898fda-c3ce-4907-a685-058d32307825-tls-certs\") pod \"model-serving-api-75bd89fdfc-prj9w\" (UID: \"c5898fda-c3ce-4907-a685-058d32307825\") " pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" Oct 30 19:27:10.330172 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.330157 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ad4552e-0da1-49db-80b8-fb4573db1d60-cert\") pod \"odh-model-controller-769875db56-795kr\" (UID: \"1ad4552e-0da1-49db-80b8-fb4573db1d60\") " pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.330330 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.330309 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad4552e-0da1-49db-80b8-fb4573db1d60-metrics-certs\") pod \"odh-model-controller-769875db56-795kr\" (UID: \"1ad4552e-0da1-49db-80b8-fb4573db1d60\") " pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.348463 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.348255 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:27:10.352803 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.352781 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bm5p\" (UniqueName: \"kubernetes.io/projected/1ad4552e-0da1-49db-80b8-fb4573db1d60-kube-api-access-5bm5p\") pod \"odh-model-controller-769875db56-795kr\" (UID: \"1ad4552e-0da1-49db-80b8-fb4573db1d60\") " pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.352951 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.352930 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5nt\" (UniqueName: \"kubernetes.io/projected/c5898fda-c3ce-4907-a685-058d32307825-kube-api-access-cs5nt\") pod \"model-serving-api-75bd89fdfc-prj9w\" (UID: \"c5898fda-c3ce-4907-a685-058d32307825\") " pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" Oct 30 19:27:10.395273 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.395247 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" Oct 30 19:27:10.436345 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.436289 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:10.610829 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.610811 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9kdnq"] Oct 30 19:27:10.613629 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:27:10.613602 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0dc02fe_b7e5_419e_80af_d0c9df5b6742.slice/crio-db1a04de070c8ed2b768fa7d71b4ce2da9ee1c7030a100e74ea61c3a772fe41c WatchSource:0}: Error finding container db1a04de070c8ed2b768fa7d71b4ce2da9ee1c7030a100e74ea61c3a772fe41c: Status 404 returned error can't find the container with id db1a04de070c8ed2b768fa7d71b4ce2da9ee1c7030a100e74ea61c3a772fe41c Oct 30 19:27:10.661216 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.661185 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t"] Oct 30 19:27:10.662197 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:27:10.662165 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda72bdca4_b46b_4a16_b57a_7bbf06de541c.slice/crio-ff81086fb8850067b9b12dd7284a15381758db0dc393faa10f1ea23e145508ee WatchSource:0}: Error finding container ff81086fb8850067b9b12dd7284a15381758db0dc393faa10f1ea23e145508ee: Status 404 returned error can't find the container with id ff81086fb8850067b9b12dd7284a15381758db0dc393faa10f1ea23e145508ee Oct 30 19:27:10.709663 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.709604 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-prj9w"] Oct 30 19:27:10.713241 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:27:10.713217 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5898fda_c3ce_4907_a685_058d32307825.slice/crio-a9e4b3427cc3d72b86620d9a9feba0093e2bc67357d721b0719774b5d0b91d86 WatchSource:0}: Error finding container a9e4b3427cc3d72b86620d9a9feba0093e2bc67357d721b0719774b5d0b91d86: Status 404 returned error can't find the container with id a9e4b3427cc3d72b86620d9a9feba0093e2bc67357d721b0719774b5d0b91d86 Oct 30 19:27:10.746333 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.746314 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-795kr"] Oct 30 19:27:10.748001 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:27:10.747977 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad4552e_0da1_49db_80b8_fb4573db1d60.slice/crio-60982ef8a96ca54046d15bd6ba8de8bdfeddb31d2412cb5326157f47c2240111 WatchSource:0}: Error finding container 60982ef8a96ca54046d15bd6ba8de8bdfeddb31d2412cb5326157f47c2240111: Status 404 returned error can't find the container with id 60982ef8a96ca54046d15bd6ba8de8bdfeddb31d2412cb5326157f47c2240111 Oct 30 19:27:10.817427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.817396 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-795kr" event={"ID":"1ad4552e-0da1-49db-80b8-fb4573db1d60","Type":"ContainerStarted","Data":"60982ef8a96ca54046d15bd6ba8de8bdfeddb31d2412cb5326157f47c2240111"} Oct 30 19:27:10.818673 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.818646 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" event={"ID":"c5898fda-c3ce-4907-a685-058d32307825","Type":"ContainerStarted","Data":"a9e4b3427cc3d72b86620d9a9feba0093e2bc67357d721b0719774b5d0b91d86"} Oct 30 19:27:10.819874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.819849 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" event={"ID":"b0dc02fe-b7e5-419e-80af-d0c9df5b6742","Type":"ContainerStarted","Data":"db1a04de070c8ed2b768fa7d71b4ce2da9ee1c7030a100e74ea61c3a772fe41c"} Oct 30 19:27:10.821023 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:10.821002 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" event={"ID":"a72bdca4-b46b-4a16-b57a-7bbf06de541c","Type":"ContainerStarted","Data":"ff81086fb8850067b9b12dd7284a15381758db0dc393faa10f1ea23e145508ee"} Oct 30 19:27:13.847347 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:13.847309 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" event={"ID":"b0dc02fe-b7e5-419e-80af-d0c9df5b6742","Type":"ContainerStarted","Data":"60dddaa34a6159013ece14b70c96028d1ddc76a4218ba5d8334ab40c69c69cd2"} Oct 30 19:27:13.847788 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:13.847408 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" Oct 30 19:27:13.849309 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:13.849279 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" event={"ID":"a72bdca4-b46b-4a16-b57a-7bbf06de541c","Type":"ContainerStarted","Data":"495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39"} Oct 30 19:27:13.849789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:13.849768 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:27:13.851643 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:13.851620 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" event={"ID":"c5898fda-c3ce-4907-a685-058d32307825","Type":"ContainerStarted","Data":"5c9731f03e0156d04e3c7e220d710141cf1f3fff93263e0c0fc48ec1426f8436"} Oct 30 19:27:13.851758 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:13.851711 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" Oct 30 19:27:13.891675 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:13.891598 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" podStartSLOduration=2.074301835 podStartE2EDuration="4.891585165s" totalImagesPullingTime="2.81728333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:13.890233874 +0000 UTC m=+394.803838493" watchObservedRunningTime="2026-10-30 19:27:13.891585165 +0000 UTC m=+394.805189781" Oct 30 19:27:13.961084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:13.961030 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" podStartSLOduration=1.005942999 podStartE2EDuration="3.961012989s" totalImagesPullingTime="2.95506999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:13.95691732 +0000 UTC m=+394.870521962" watchObservedRunningTime="2026-10-30 19:27:13.961012989 +0000 UTC m=+394.874617608" Oct 30 19:27:14.034449 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:14.034385 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" podStartSLOduration=1.441220474 podStartE2EDuration="4.034368063s" totalImagesPullingTime="2.593147589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:14.033449091 +0000 UTC m=+394.947053711" watchObservedRunningTime="2026-10-30 19:27:14.034368063 +0000 UTC m=+394.947972679" Oct 30 19:27:14.858833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:14.858785 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-795kr" event={"ID":"1ad4552e-0da1-49db-80b8-fb4573db1d60","Type":"ContainerStarted","Data":"9bd20ef60a79fc522973914a20da6601e4a5824119d47ecb4aaecc37d15b882b"} Oct 30 19:27:14.859300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:14.858990 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:14.901087 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:14.901046 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-795kr" podStartSLOduration=1.444015513 podStartE2EDuration="4.901034032s" totalImagesPullingTime="3.457018519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:14.900778141 +0000 UTC m=+395.814382770" watchObservedRunningTime="2026-10-30 19:27:14.901034032 +0000 UTC m=+395.814638648" Oct 30 19:27:24.508938 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:27:24.508862 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:27:24.862898 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:24.862834 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-prj9w" Oct 30 19:27:25.867935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:25.867908 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-795kr" Oct 30 19:27:38.164899 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.164859 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rq5hx"] Oct 30 19:27:38.211421 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.211390 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rq5hx"] Oct 30 19:27:38.211587 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.211433 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" Oct 30 19:27:38.282273 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.282236 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jspw\" (UniqueName: \"kubernetes.io/projected/41724e5c-4629-4e87-a018-c8dac5320087-kube-api-access-9jspw\") pod \"authorino-6f6944bcdd-rq5hx\" (UID: \"41724e5c-4629-4e87-a018-c8dac5320087\") " pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" Oct 30 19:27:38.383159 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.383129 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jspw\" (UniqueName: \"kubernetes.io/projected/41724e5c-4629-4e87-a018-c8dac5320087-kube-api-access-9jspw\") pod \"authorino-6f6944bcdd-rq5hx\" (UID: \"41724e5c-4629-4e87-a018-c8dac5320087\") " pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" Oct 30 19:27:38.411984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.411956 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jspw\" (UniqueName: \"kubernetes.io/projected/41724e5c-4629-4e87-a018-c8dac5320087-kube-api-access-9jspw\") pod \"authorino-6f6944bcdd-rq5hx\" (UID: \"41724e5c-4629-4e87-a018-c8dac5320087\") " pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" Oct 30 19:27:38.455906 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.455878 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rq5hx"] Oct 30 19:27:38.456149 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.456133 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" Oct 30 19:27:38.503303 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.503270 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vr495"] Oct 30 19:27:38.535382 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.532755 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-vr495" Oct 30 19:27:38.539930 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.539875 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vr495"] Oct 30 19:27:38.554524 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.554482 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 19:27:38.567902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.567875 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vr495"] Oct 30 19:27:38.568170 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:27:38.568146 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7pxmm tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-7pxmm tls-cert]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-vr495" podUID="77aa02a7-c5b4-4c39-a644-74f6455ccdc5" Oct 30 19:27:38.584738 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.584715 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pxmm\" (UniqueName: \"kubernetes.io/projected/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-kube-api-access-7pxmm\") pod \"authorino-6889cbd8c4-vr495\" (UID: \"77aa02a7-c5b4-4c39-a644-74f6455ccdc5\") " pod="kuadrant-system/authorino-6889cbd8c4-vr495" Oct 30 19:27:38.584841 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.584756 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-tls-cert\") pod \"authorino-6889cbd8c4-vr495\" (UID: \"77aa02a7-c5b4-4c39-a644-74f6455ccdc5\") " pod="kuadrant-system/authorino-6889cbd8c4-vr495" Oct 30 19:27:38.604048 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.604026 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b9dfc9f89-vwmxj"] Oct 30 19:27:38.620230 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.620204 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ggdsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:38.620380 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.620243 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" podUID="0c8a2a4b-6240-40fa-a873-d2300e317a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:38.625594 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.625571 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" Oct 30 19:27:38.628549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.628524 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b9dfc9f89-vwmxj"] Oct 30 19:27:38.686252 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.686220 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-tls-cert\") pod \"authorino-6889cbd8c4-vr495\" (UID: \"77aa02a7-c5b4-4c39-a644-74f6455ccdc5\") " pod="kuadrant-system/authorino-6889cbd8c4-vr495" Oct 30 19:27:38.686380 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.686298 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zrn\" (UniqueName: \"kubernetes.io/projected/41fe341a-033a-41ec-8fcb-6f86e868c777-kube-api-access-b9zrn\") pod \"authorino-5b9dfc9f89-vwmxj\" (UID: \"41fe341a-033a-41ec-8fcb-6f86e868c777\") " pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" Oct 30 19:27:38.686450 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.686386 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41fe341a-033a-41ec-8fcb-6f86e868c777-tls-cert\") pod \"authorino-5b9dfc9f89-vwmxj\" (UID: \"41fe341a-033a-41ec-8fcb-6f86e868c777\") " pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" Oct 30 19:27:38.686506 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.686473 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pxmm\" (UniqueName: \"kubernetes.io/projected/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-kube-api-access-7pxmm\") pod \"authorino-6889cbd8c4-vr495\" (UID: \"77aa02a7-c5b4-4c39-a644-74f6455ccdc5\") " pod="kuadrant-system/authorino-6889cbd8c4-vr495" Oct 30 19:27:38.689017 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.688998 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-tls-cert\") pod \"authorino-6889cbd8c4-vr495\" (UID: \"77aa02a7-c5b4-4c39-a644-74f6455ccdc5\") " pod="kuadrant-system/authorino-6889cbd8c4-vr495" Oct 30 19:27:38.707978 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.707926 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pxmm\" (UniqueName: \"kubernetes.io/projected/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-kube-api-access-7pxmm\") pod \"authorino-6889cbd8c4-vr495\" (UID: \"77aa02a7-c5b4-4c39-a644-74f6455ccdc5\") " pod="kuadrant-system/authorino-6889cbd8c4-vr495" Oct 30 19:27:38.771987 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.771966 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rq5hx"] Oct 30 19:27:38.774706 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:27:38.774670 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41724e5c_4629_4e87_a018_c8dac5320087.slice/crio-e533d6db931cbf4fbc1a11a4676a5eed23d6a4808966326ca6782b9e89c35bce WatchSource:0}: Error finding container e533d6db931cbf4fbc1a11a4676a5eed23d6a4808966326ca6782b9e89c35bce: Status 404 returned error can't find the container with id e533d6db931cbf4fbc1a11a4676a5eed23d6a4808966326ca6782b9e89c35bce Oct 30 19:27:38.787017 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.786998 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zrn\" (UniqueName: \"kubernetes.io/projected/41fe341a-033a-41ec-8fcb-6f86e868c777-kube-api-access-b9zrn\") pod \"authorino-5b9dfc9f89-vwmxj\" (UID: \"41fe341a-033a-41ec-8fcb-6f86e868c777\") " pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" Oct 30 19:27:38.787096 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.787028 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41fe341a-033a-41ec-8fcb-6f86e868c777-tls-cert\") pod \"authorino-5b9dfc9f89-vwmxj\" (UID: \"41fe341a-033a-41ec-8fcb-6f86e868c777\") " pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" Oct 30 19:27:38.789803 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.789763 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41fe341a-033a-41ec-8fcb-6f86e868c777-tls-cert\") pod \"authorino-5b9dfc9f89-vwmxj\" (UID: \"41fe341a-033a-41ec-8fcb-6f86e868c777\") " pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" Oct 30 19:27:38.808596 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.808572 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zrn\" (UniqueName: \"kubernetes.io/projected/41fe341a-033a-41ec-8fcb-6f86e868c777-kube-api-access-b9zrn\") pod \"authorino-5b9dfc9f89-vwmxj\" (UID: \"41fe341a-033a-41ec-8fcb-6f86e868c777\") " pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" Oct 30 19:27:38.952350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:38.952324 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" Oct 30 19:27:39.005054 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:39.005021 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" event={"ID":"41724e5c-4629-4e87-a018-c8dac5320087","Type":"ContainerStarted","Data":"e533d6db931cbf4fbc1a11a4676a5eed23d6a4808966326ca6782b9e89c35bce"} Oct 30 19:27:39.005198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:39.005149 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-vr495" Oct 30 19:27:39.031533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:39.031511 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-vr495" Oct 30 19:27:39.090041 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:39.089812 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-tls-cert\" (UniqueName: \"kubernetes.io/secret/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-tls-cert\") pod \"77aa02a7-c5b4-4c39-a644-74f6455ccdc5\" (UID: \"77aa02a7-c5b4-4c39-a644-74f6455ccdc5\") " Oct 30 19:27:39.090041 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:39.089885 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-kube-api-access-7pxmm\" (UniqueName: \"kubernetes.io/projected/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-kube-api-access-7pxmm\") pod \"77aa02a7-c5b4-4c39-a644-74f6455ccdc5\" (UID: \"77aa02a7-c5b4-4c39-a644-74f6455ccdc5\") " Oct 30 19:27:39.092178 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:39.092148 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-kube-api-access-7pxmm" pod "77aa02a7-c5b4-4c39-a644-74f6455ccdc5" (UID: "77aa02a7-c5b4-4c39-a644-74f6455ccdc5"). InnerVolumeSpecName "kube-api-access-7pxmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:39.092665 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:39.092649 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-tls-cert" pod "77aa02a7-c5b4-4c39-a644-74f6455ccdc5" (UID: "77aa02a7-c5b4-4c39-a644-74f6455ccdc5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:39.190543 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:39.190514 4904 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-tls-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:27:39.190543 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:39.190541 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pxmm\" (UniqueName: \"kubernetes.io/projected/77aa02a7-c5b4-4c39-a644-74f6455ccdc5-kube-api-access-7pxmm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:27:39.284647 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:39.284623 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b9dfc9f89-vwmxj"] Oct 30 19:27:39.286369 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:27:39.286341 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41fe341a_033a_41ec_8fcb_6f86e868c777.slice/crio-c4a5c473b80b88ebed500c32539345426ff035c338f3c08b0c2a1a2f8790834c WatchSource:0}: Error finding container c4a5c473b80b88ebed500c32539345426ff035c338f3c08b0c2a1a2f8790834c: Status 404 returned error can't find the container with id c4a5c473b80b88ebed500c32539345426ff035c338f3c08b0c2a1a2f8790834c Oct 30 19:27:40.011745 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:40.011707 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" event={"ID":"41fe341a-033a-41ec-8fcb-6f86e868c777","Type":"ContainerStarted","Data":"c4a5c473b80b88ebed500c32539345426ff035c338f3c08b0c2a1a2f8790834c"} Oct 30 19:27:40.011895 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:40.011792 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-vr495" Oct 30 19:27:40.090280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:40.090250 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vr495"] Oct 30 19:27:40.093082 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:40.093060 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vr495"] Oct 30 19:27:41.027371 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.027337 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" event={"ID":"41fe341a-033a-41ec-8fcb-6f86e868c777","Type":"ContainerStarted","Data":"0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5"} Oct 30 19:27:41.029055 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.029027 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" event={"ID":"41724e5c-4629-4e87-a018-c8dac5320087","Type":"ContainerStarted","Data":"4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11"} Oct 30 19:27:41.029190 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.029051 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" podUID="41724e5c-4629-4e87-a018-c8dac5320087" containerName="authorino" containerID="cri-o://4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11" gracePeriod=30 Oct 30 19:27:41.030640 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.030616 4904 generic.go:356] "Generic (PLEG): container finished" podID="0c8a2a4b-6240-40fa-a873-d2300e317a2b" containerID="96668163d9dee1f37e1451e6e6a8c0b37f140edf9445dc51b304cc447019d47e" exitCode=0 Oct 30 19:27:41.030744 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.030672 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" event={"ID":"0c8a2a4b-6240-40fa-a873-d2300e317a2b","Type":"ContainerDied","Data":"96668163d9dee1f37e1451e6e6a8c0b37f140edf9445dc51b304cc447019d47e"} Oct 30 19:27:41.030982 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.030967 4904 scope.go:122] "RemoveContainer" containerID="96668163d9dee1f37e1451e6e6a8c0b37f140edf9445dc51b304cc447019d47e" Oct 30 19:27:41.032216 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.032180 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:27:41.032309 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.032278 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:27:41.032354 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.032323 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:27:41.074864 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.074826 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" podStartSLOduration=2.3249460490000002 podStartE2EDuration="3.07481549s" totalImagesPullingTime="749.869441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:41.072734594 +0000 UTC m=+421.986339211" watchObservedRunningTime="2026-10-30 19:27:41.07481549 +0000 UTC m=+421.988420108" Oct 30 19:27:41.109977 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.109949 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h4fdc"] Oct 30 19:27:41.110145 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.110125 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" podUID="999de8b6-2491-49dd-9008-b659915aeca4" containerName="authorino" containerID="cri-o://0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2" gracePeriod=30 Oct 30 19:27:41.180624 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.180572 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" podStartSLOduration=1.939246464 podStartE2EDuration="3.180558388s" totalImagesPullingTime="1.241311924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:41.129690109 +0000 UTC m=+422.043294726" watchObservedRunningTime="2026-10-30 19:27:41.180558388 +0000 UTC m=+422.094163041" Oct 30 19:27:41.227448 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.225310 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77aa02a7-c5b4-4c39-a644-74f6455ccdc5" path="/var/lib/kubelet/pods/77aa02a7-c5b4-4c39-a644-74f6455ccdc5/volumes" Oct 30 19:27:41.393331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.393307 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" Oct 30 19:27:41.408388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.408362 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" Oct 30 19:27:41.512576 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.512546 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999de8b6-2491-49dd-9008-b659915aeca4-kube-api-access-vqhf5\" (UniqueName: \"kubernetes.io/projected/999de8b6-2491-49dd-9008-b659915aeca4-kube-api-access-vqhf5\") pod \"999de8b6-2491-49dd-9008-b659915aeca4\" (UID: \"999de8b6-2491-49dd-9008-b659915aeca4\") " Oct 30 19:27:41.512737 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.512583 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41724e5c-4629-4e87-a018-c8dac5320087-kube-api-access-9jspw\" (UniqueName: \"kubernetes.io/projected/41724e5c-4629-4e87-a018-c8dac5320087-kube-api-access-9jspw\") pod \"41724e5c-4629-4e87-a018-c8dac5320087\" (UID: \"41724e5c-4629-4e87-a018-c8dac5320087\") " Oct 30 19:27:41.515060 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.515019 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41724e5c-4629-4e87-a018-c8dac5320087-kube-api-access-9jspw" pod "41724e5c-4629-4e87-a018-c8dac5320087" (UID: "41724e5c-4629-4e87-a018-c8dac5320087"). InnerVolumeSpecName "kube-api-access-9jspw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:41.515200 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.515180 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999de8b6-2491-49dd-9008-b659915aeca4-kube-api-access-vqhf5" pod "999de8b6-2491-49dd-9008-b659915aeca4" (UID: "999de8b6-2491-49dd-9008-b659915aeca4"). InnerVolumeSpecName "kube-api-access-vqhf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:41.613544 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.613469 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqhf5\" (UniqueName: \"kubernetes.io/projected/999de8b6-2491-49dd-9008-b659915aeca4-kube-api-access-vqhf5\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:27:41.613544 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:41.613498 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jspw\" (UniqueName: \"kubernetes.io/projected/41724e5c-4629-4e87-a018-c8dac5320087-kube-api-access-9jspw\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:27:42.044147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.044076 4904 generic.go:356] "Generic (PLEG): container finished" podID="41724e5c-4629-4e87-a018-c8dac5320087" containerID="4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11" exitCode=0 Oct 30 19:27:42.044147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.044134 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" event={"ID":"41724e5c-4629-4e87-a018-c8dac5320087","Type":"ContainerDied","Data":"4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11"} Oct 30 19:27:42.044147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.044148 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" Oct 30 19:27:42.044663 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.044183 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-rq5hx" event={"ID":"41724e5c-4629-4e87-a018-c8dac5320087","Type":"ContainerDied","Data":"e533d6db931cbf4fbc1a11a4676a5eed23d6a4808966326ca6782b9e89c35bce"} Oct 30 19:27:42.044663 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.044205 4904 scope.go:122] "RemoveContainer" containerID="4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11" Oct 30 19:27:42.046852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.046824 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" event={"ID":"0c8a2a4b-6240-40fa-a873-d2300e317a2b","Type":"ContainerStarted","Data":"72bcec47d288401254078f2b6cab4d22ccbe7d9f90f3a2ad34edf098ed3353cc"} Oct 30 19:27:42.048081 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.048062 4904 generic.go:356] "Generic (PLEG): container finished" podID="999de8b6-2491-49dd-9008-b659915aeca4" containerID="0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2" exitCode=0 Oct 30 19:27:42.048205 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.048140 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" event={"ID":"999de8b6-2491-49dd-9008-b659915aeca4","Type":"ContainerDied","Data":"0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2"} Oct 30 19:27:42.048205 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.048175 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" event={"ID":"999de8b6-2491-49dd-9008-b659915aeca4","Type":"ContainerDied","Data":"c844af3fe9f640953e9dd55599bc2a4c2a524533cb7c6549343d736add08e697"} Oct 30 19:27:42.048205 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.048145 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h4fdc" Oct 30 19:27:42.055320 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.055303 4904 scope.go:122] "RemoveContainer" containerID="4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11" Oct 30 19:27:42.055590 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:27:42.055572 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11\": container with ID starting with 4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11 not found: ID does not exist" containerID="4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11" Oct 30 19:27:42.055650 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.055596 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11"} err="failed to get container status \"4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11\": rpc error: code = NotFound desc = could not find container \"4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11\": container with ID starting with 4b665cb86cd7a097ee8b9ee7ac2cc3b56f1f64b834e44ffb619e57ce62a43f11 not found: ID does not exist" Oct 30 19:27:42.055650 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.055613 4904 scope.go:122] "RemoveContainer" containerID="0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2" Oct 30 19:27:42.064038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.064019 4904 scope.go:122] "RemoveContainer" containerID="0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2" Oct 30 19:27:42.064285 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:27:42.064267 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2\": container with ID starting with 0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2 not found: ID does not exist" containerID="0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2" Oct 30 19:27:42.064359 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.064290 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2"} err="failed to get container status \"0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2\": rpc error: code = NotFound desc = could not find container \"0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2\": container with ID starting with 0e931509f2198bf9bcf63d7e80857386bda8b16b3b88ef3055314407fa0ac8d2 not found: ID does not exist" Oct 30 19:27:42.099520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.099497 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rq5hx"] Oct 30 19:27:42.105824 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.105807 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rq5hx"] Oct 30 19:27:42.208372 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.208333 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h4fdc"] Oct 30 19:27:42.239128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.235788 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h4fdc"] Oct 30 19:27:42.766887 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.766858 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:27:42.767241 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.767217 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ggdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 19:27:42.767369 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:42.767266 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" podUID="0c8a2a4b-6240-40fa-a873-d2300e317a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 19:27:43.241081 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:43.241046 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41724e5c-4629-4e87-a018-c8dac5320087" path="/var/lib/kubelet/pods/41724e5c-4629-4e87-a018-c8dac5320087/volumes" Oct 30 19:27:43.241500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:43.241391 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999de8b6-2491-49dd-9008-b659915aeca4" path="/var/lib/kubelet/pods/999de8b6-2491-49dd-9008-b659915aeca4/volumes" Oct 30 19:27:43.767534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:43.767492 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ggdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 19:27:43.767689 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:43.767546 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" podUID="0c8a2a4b-6240-40fa-a873-d2300e317a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 19:27:44.767772 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:44.767730 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ggdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 19:27:44.768167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:44.767806 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" podUID="0c8a2a4b-6240-40fa-a873-d2300e317a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 19:27:44.864867 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:44.864837 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9kdnq" Oct 30 19:27:45.767358 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:45.767324 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ggdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 19:27:45.767518 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:45.767380 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" podUID="0c8a2a4b-6240-40fa-a873-d2300e317a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 19:27:45.867370 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:45.867346 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:27:46.767385 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:46.767339 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ggdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 19:27:46.767606 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:46.767416 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" podUID="0c8a2a4b-6240-40fa-a873-d2300e317a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 19:27:47.770706 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:47.770678 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:27:47.771119 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:47.770952 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:27:47.771661 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:27:47.771645 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" Oct 30 19:27:54.510719 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:27:54.510684 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:28:22.446160 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.446126 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b"] Oct 30 19:28:22.446656 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.446550 4904 cpu_manager.go:414] "containerMap: removing container" podUID="41724e5c-4629-4e87-a018-c8dac5320087" containerName="authorino" Oct 30 19:28:22.446656 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.446561 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41724e5c-4629-4e87-a018-c8dac5320087" containerName="authorino" Oct 30 19:28:22.446656 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.446578 4904 cpu_manager.go:414] "containerMap: removing container" podUID="999de8b6-2491-49dd-9008-b659915aeca4" containerName="authorino" Oct 30 19:28:22.446656 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.446584 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999de8b6-2491-49dd-9008-b659915aeca4" containerName="authorino" Oct 30 19:28:22.446782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.446674 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="41724e5c-4629-4e87-a018-c8dac5320087" containerName="authorino" Oct 30 19:28:22.446782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.446681 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="999de8b6-2491-49dd-9008-b659915aeca4" containerName="authorino" Oct 30 19:28:22.449843 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.449826 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" Oct 30 19:28:22.466857 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.466834 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b"] Oct 30 19:28:22.576653 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.576612 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtqs\" (UniqueName: \"kubernetes.io/projected/c9e60db3-d94e-40a8-9ad0-8abe2148fe84-kube-api-access-9rtqs\") pod \"llmisvc-controller-manager-6466c554d8-4lh4b\" (UID: \"c9e60db3-d94e-40a8-9ad0-8abe2148fe84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" Oct 30 19:28:22.576835 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.576762 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9e60db3-d94e-40a8-9ad0-8abe2148fe84-cert\") pod \"llmisvc-controller-manager-6466c554d8-4lh4b\" (UID: \"c9e60db3-d94e-40a8-9ad0-8abe2148fe84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" Oct 30 19:28:22.678094 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.678052 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9e60db3-d94e-40a8-9ad0-8abe2148fe84-cert\") pod \"llmisvc-controller-manager-6466c554d8-4lh4b\" (UID: \"c9e60db3-d94e-40a8-9ad0-8abe2148fe84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" Oct 30 19:28:22.678316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.678153 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtqs\" (UniqueName: \"kubernetes.io/projected/c9e60db3-d94e-40a8-9ad0-8abe2148fe84-kube-api-access-9rtqs\") pod \"llmisvc-controller-manager-6466c554d8-4lh4b\" (UID: \"c9e60db3-d94e-40a8-9ad0-8abe2148fe84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" Oct 30 19:28:22.680681 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.680652 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9e60db3-d94e-40a8-9ad0-8abe2148fe84-cert\") pod \"llmisvc-controller-manager-6466c554d8-4lh4b\" (UID: \"c9e60db3-d94e-40a8-9ad0-8abe2148fe84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" Oct 30 19:28:22.699520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.699458 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtqs\" (UniqueName: \"kubernetes.io/projected/c9e60db3-d94e-40a8-9ad0-8abe2148fe84-kube-api-access-9rtqs\") pod \"llmisvc-controller-manager-6466c554d8-4lh4b\" (UID: \"c9e60db3-d94e-40a8-9ad0-8abe2148fe84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" Oct 30 19:28:22.763279 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:22.763244 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" Oct 30 19:28:23.064438 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:23.064402 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b"] Oct 30 19:28:23.066313 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:28:23.066283 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e60db3_d94e_40a8_9ad0_8abe2148fe84.slice/crio-833bd64170727c9ec1d67aa599fa6c52d1b0f3f228b0f09405d0e69701948ac6 WatchSource:0}: Error finding container 833bd64170727c9ec1d67aa599fa6c52d1b0f3f228b0f09405d0e69701948ac6: Status 404 returned error can't find the container with id 833bd64170727c9ec1d67aa599fa6c52d1b0f3f228b0f09405d0e69701948ac6 Oct 30 19:28:23.280300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:23.280259 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" event={"ID":"c9e60db3-d94e-40a8-9ad0-8abe2148fe84","Type":"ContainerStarted","Data":"833bd64170727c9ec1d67aa599fa6c52d1b0f3f228b0f09405d0e69701948ac6"} Oct 30 19:28:24.288366 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:24.288329 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" event={"ID":"c9e60db3-d94e-40a8-9ad0-8abe2148fe84","Type":"ContainerStarted","Data":"1f0b366b51d1efcd841f18f1fcde9ad0a01dd674127b36960cddf84ac5a94ef3"} Oct 30 19:28:24.288878 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:24.288452 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" Oct 30 19:28:24.330487 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:24.330443 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" podStartSLOduration=2.015505595 podStartE2EDuration="2.330430366s" totalImagesPullingTime="314.924771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:24.32912971 +0000 UTC m=+465.242734328" watchObservedRunningTime="2026-10-30 19:28:24.330430366 +0000 UTC m=+465.244034982" Oct 30 19:28:24.508877 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:28:24.508842 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:28:54.509006 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:28:54.508926 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:28:55.297217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:55.297190 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4lh4b" Oct 30 19:28:55.370629 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:55.370587 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t"] Oct 30 19:28:55.370835 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:55.370799 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" podUID="a72bdca4-b46b-4a16-b57a-7bbf06de541c" containerName="manager" containerID="cri-o://495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39" gracePeriod=30 Oct 30 19:28:55.634989 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:55.634962 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:28:55.652158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:55.652132 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72bdca4-b46b-4a16-b57a-7bbf06de541c-cert\" (UniqueName: \"kubernetes.io/secret/a72bdca4-b46b-4a16-b57a-7bbf06de541c-cert\") pod \"a72bdca4-b46b-4a16-b57a-7bbf06de541c\" (UID: \"a72bdca4-b46b-4a16-b57a-7bbf06de541c\") " Oct 30 19:28:55.652269 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:55.652194 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72bdca4-b46b-4a16-b57a-7bbf06de541c-kube-api-access-ddnst\" (UniqueName: \"kubernetes.io/projected/a72bdca4-b46b-4a16-b57a-7bbf06de541c-kube-api-access-ddnst\") pod \"a72bdca4-b46b-4a16-b57a-7bbf06de541c\" (UID: \"a72bdca4-b46b-4a16-b57a-7bbf06de541c\") " Oct 30 19:28:55.654478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:55.654454 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72bdca4-b46b-4a16-b57a-7bbf06de541c-kube-api-access-ddnst" pod "a72bdca4-b46b-4a16-b57a-7bbf06de541c" (UID: "a72bdca4-b46b-4a16-b57a-7bbf06de541c"). InnerVolumeSpecName "kube-api-access-ddnst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:55.654583 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:55.654556 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72bdca4-b46b-4a16-b57a-7bbf06de541c-cert" pod "a72bdca4-b46b-4a16-b57a-7bbf06de541c" (UID: "a72bdca4-b46b-4a16-b57a-7bbf06de541c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:55.753212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:55.753186 4904 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a72bdca4-b46b-4a16-b57a-7bbf06de541c-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:28:55.753212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:55.753211 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddnst\" (UniqueName: \"kubernetes.io/projected/a72bdca4-b46b-4a16-b57a-7bbf06de541c-kube-api-access-ddnst\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:28:56.467583 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:56.467549 4904 generic.go:356] "Generic (PLEG): container finished" podID="a72bdca4-b46b-4a16-b57a-7bbf06de541c" containerID="495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39" exitCode=0 Oct 30 19:28:56.467776 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:56.467593 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" Oct 30 19:28:56.467776 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:56.467634 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" event={"ID":"a72bdca4-b46b-4a16-b57a-7bbf06de541c","Type":"ContainerDied","Data":"495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39"} Oct 30 19:28:56.467776 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:56.467698 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t" event={"ID":"a72bdca4-b46b-4a16-b57a-7bbf06de541c","Type":"ContainerDied","Data":"ff81086fb8850067b9b12dd7284a15381758db0dc393faa10f1ea23e145508ee"} Oct 30 19:28:56.467776 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:56.467729 4904 scope.go:122] "RemoveContainer" containerID="495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39" Oct 30 19:28:56.478587 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:56.478568 4904 scope.go:122] "RemoveContainer" containerID="495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39" Oct 30 19:28:56.478831 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:28:56.478812 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39\": container with ID starting with 495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39 not found: ID does not exist" containerID="495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39" Oct 30 19:28:56.478892 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:56.478841 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39"} err="failed to get container status \"495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39\": rpc error: code = NotFound desc = could not find container \"495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39\": container with ID starting with 495287f6339f4ba315e7e8b7892d41d5d034238d897e9fc3a9684fab3171ca39 not found: ID does not exist" Oct 30 19:28:56.530339 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:56.530321 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t"] Oct 30 19:28:56.543713 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:56.543685 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zgf8t"] Oct 30 19:28:57.221270 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:28:57.221238 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72bdca4-b46b-4a16-b57a-7bbf06de541c" path="/var/lib/kubelet/pods/a72bdca4-b46b-4a16-b57a-7bbf06de541c/volumes" Oct 30 19:29:24.511088 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:29:24.511047 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:29:44.054413 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.054381 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54554668db-6fwsb"] Oct 30 19:29:44.054843 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.054770 4904 cpu_manager.go:414] "containerMap: removing container" podUID="a72bdca4-b46b-4a16-b57a-7bbf06de541c" containerName="manager" Oct 30 19:29:44.054843 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.054781 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72bdca4-b46b-4a16-b57a-7bbf06de541c" containerName="manager" Oct 30 19:29:44.054922 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.054860 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72bdca4-b46b-4a16-b57a-7bbf06de541c" containerName="manager" Oct 30 19:29:44.059336 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.059315 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54554668db-6fwsb" Oct 30 19:29:44.072854 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.072830 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54554668db-6fwsb"] Oct 30 19:29:44.196151 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.196119 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkk5v\" (UniqueName: \"kubernetes.io/projected/9ead0538-3773-4680-b10f-038113f65fcf-kube-api-access-tkk5v\") pod \"authorino-54554668db-6fwsb\" (UID: \"9ead0538-3773-4680-b10f-038113f65fcf\") " pod="kuadrant-system/authorino-54554668db-6fwsb" Oct 30 19:29:44.196296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.196182 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ead0538-3773-4680-b10f-038113f65fcf-tls-cert\") pod \"authorino-54554668db-6fwsb\" (UID: \"9ead0538-3773-4680-b10f-038113f65fcf\") " pod="kuadrant-system/authorino-54554668db-6fwsb" Oct 30 19:29:44.296738 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.296707 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ead0538-3773-4680-b10f-038113f65fcf-tls-cert\") pod \"authorino-54554668db-6fwsb\" (UID: \"9ead0538-3773-4680-b10f-038113f65fcf\") " pod="kuadrant-system/authorino-54554668db-6fwsb" Oct 30 19:29:44.296938 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.296908 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkk5v\" (UniqueName: \"kubernetes.io/projected/9ead0538-3773-4680-b10f-038113f65fcf-kube-api-access-tkk5v\") pod \"authorino-54554668db-6fwsb\" (UID: \"9ead0538-3773-4680-b10f-038113f65fcf\") " pod="kuadrant-system/authorino-54554668db-6fwsb" Oct 30 19:29:44.299442 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.299417 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ead0538-3773-4680-b10f-038113f65fcf-tls-cert\") pod \"authorino-54554668db-6fwsb\" (UID: \"9ead0538-3773-4680-b10f-038113f65fcf\") " pod="kuadrant-system/authorino-54554668db-6fwsb" Oct 30 19:29:44.317846 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.317784 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkk5v\" (UniqueName: \"kubernetes.io/projected/9ead0538-3773-4680-b10f-038113f65fcf-kube-api-access-tkk5v\") pod \"authorino-54554668db-6fwsb\" (UID: \"9ead0538-3773-4680-b10f-038113f65fcf\") " pod="kuadrant-system/authorino-54554668db-6fwsb" Oct 30 19:29:44.372235 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.372210 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54554668db-6fwsb" Oct 30 19:29:44.672032 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.672003 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54554668db-6fwsb"] Oct 30 19:29:44.673578 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:29:44.673544 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ead0538_3773_4680_b10f_038113f65fcf.slice/crio-65580f11b7b243e08914e031c8d81c774da19266a55e55d07d41a2f7d0054050 WatchSource:0}: Error finding container 65580f11b7b243e08914e031c8d81c774da19266a55e55d07d41a2f7d0054050: Status 404 returned error can't find the container with id 65580f11b7b243e08914e031c8d81c774da19266a55e55d07d41a2f7d0054050 Oct 30 19:29:44.770774 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:44.770735 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54554668db-6fwsb" event={"ID":"9ead0538-3773-4680-b10f-038113f65fcf","Type":"ContainerStarted","Data":"65580f11b7b243e08914e031c8d81c774da19266a55e55d07d41a2f7d0054050"} Oct 30 19:29:46.788440 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:46.788392 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54554668db-6fwsb" event={"ID":"9ead0538-3773-4680-b10f-038113f65fcf","Type":"ContainerStarted","Data":"3cb7a217a5abcd65f34766decca4258c79658db730c8b8824273e8b9c0767fc3"} Oct 30 19:29:46.832475 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:46.832410 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-54554668db-6fwsb" podStartSLOduration=1.2082785440000001 podStartE2EDuration="2.832392851s" totalImagesPullingTime="1.624114307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:46.829949292 +0000 UTC m=+547.743553910" watchObservedRunningTime="2026-10-30 19:29:46.832392851 +0000 UTC m=+547.745997470" Oct 30 19:29:46.866600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:46.866567 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b9dfc9f89-vwmxj"] Oct 30 19:29:46.866807 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:46.866780 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" podUID="41fe341a-033a-41ec-8fcb-6f86e868c777" containerName="authorino" containerID="cri-o://0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5" gracePeriod=30 Oct 30 19:29:47.142257 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.142226 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" Oct 30 19:29:47.324260 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.324190 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41fe341a-033a-41ec-8fcb-6f86e868c777-kube-api-access-b9zrn\" (UniqueName: \"kubernetes.io/projected/41fe341a-033a-41ec-8fcb-6f86e868c777-kube-api-access-b9zrn\") pod \"41fe341a-033a-41ec-8fcb-6f86e868c777\" (UID: \"41fe341a-033a-41ec-8fcb-6f86e868c777\") " Oct 30 19:29:47.324260 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.324222 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41fe341a-033a-41ec-8fcb-6f86e868c777-tls-cert\" (UniqueName: \"kubernetes.io/secret/41fe341a-033a-41ec-8fcb-6f86e868c777-tls-cert\") pod \"41fe341a-033a-41ec-8fcb-6f86e868c777\" (UID: \"41fe341a-033a-41ec-8fcb-6f86e868c777\") " Oct 30 19:29:47.326475 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.326450 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fe341a-033a-41ec-8fcb-6f86e868c777-kube-api-access-b9zrn" pod "41fe341a-033a-41ec-8fcb-6f86e868c777" (UID: "41fe341a-033a-41ec-8fcb-6f86e868c777"). InnerVolumeSpecName "kube-api-access-b9zrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:47.335583 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.335566 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41fe341a-033a-41ec-8fcb-6f86e868c777-tls-cert" pod "41fe341a-033a-41ec-8fcb-6f86e868c777" (UID: "41fe341a-033a-41ec-8fcb-6f86e868c777"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:47.426676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.426610 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9zrn\" (UniqueName: \"kubernetes.io/projected/41fe341a-033a-41ec-8fcb-6f86e868c777-kube-api-access-b9zrn\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:29:47.426676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.426647 4904 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41fe341a-033a-41ec-8fcb-6f86e868c777-tls-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:29:47.795797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.795758 4904 generic.go:356] "Generic (PLEG): container finished" podID="41fe341a-033a-41ec-8fcb-6f86e868c777" containerID="0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5" exitCode=0 Oct 30 19:29:47.796339 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.795857 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" event={"ID":"41fe341a-033a-41ec-8fcb-6f86e868c777","Type":"ContainerDied","Data":"0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5"} Oct 30 19:29:47.796339 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.795890 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" Oct 30 19:29:47.796339 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.795916 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b9dfc9f89-vwmxj" event={"ID":"41fe341a-033a-41ec-8fcb-6f86e868c777","Type":"ContainerDied","Data":"c4a5c473b80b88ebed500c32539345426ff035c338f3c08b0c2a1a2f8790834c"} Oct 30 19:29:47.796339 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.795939 4904 scope.go:122] "RemoveContainer" containerID="0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5" Oct 30 19:29:47.808174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.808080 4904 scope.go:122] "RemoveContainer" containerID="0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5" Oct 30 19:29:47.808609 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:29:47.808582 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5\": container with ID starting with 0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5 not found: ID does not exist" containerID="0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5" Oct 30 19:29:47.808778 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.808744 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5"} err="failed to get container status \"0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5\": rpc error: code = NotFound desc = could not find container \"0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5\": container with ID starting with 0cbfb3d3a1c3298bd8e8344ec44949de6a709a98e199eef49184f2daf65438e5 not found: ID does not exist" Oct 30 19:29:47.851871 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.851844 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b9dfc9f89-vwmxj"] Oct 30 19:29:47.858498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:47.858474 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b9dfc9f89-vwmxj"] Oct 30 19:29:49.221717 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:49.221684 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41fe341a-033a-41ec-8fcb-6f86e868c777" path="/var/lib/kubelet/pods/41fe341a-033a-41ec-8fcb-6f86e868c777/volumes" Oct 30 19:29:53.323219 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.323177 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn"] Oct 30 19:29:53.323718 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.323658 4904 cpu_manager.go:414] "containerMap: removing container" podUID="41fe341a-033a-41ec-8fcb-6f86e868c777" containerName="authorino" Oct 30 19:29:53.323718 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.323676 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41fe341a-033a-41ec-8fcb-6f86e868c777" containerName="authorino" Oct 30 19:29:53.323834 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.323787 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="41fe341a-033a-41ec-8fcb-6f86e868c777" containerName="authorino" Oct 30 19:29:53.326911 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.326889 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.359450 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.359424 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn"] Oct 30 19:29:53.424379 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.424350 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.424535 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.424394 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.424535 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.424434 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.424535 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.424529 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c012900-326d-42d5-8076-3bbbdbb1e852-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.424647 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.424566 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.424647 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.424592 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.424713 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.424666 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.424713 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.424685 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmsq\" (UniqueName: \"kubernetes.io/projected/2c012900-326d-42d5-8076-3bbbdbb1e852-kube-api-access-7hmsq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.424713 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.424707 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526091 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526049 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526123 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c012900-326d-42d5-8076-3bbbdbb1e852-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526162 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526197 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526242 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526267 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmsq\" (UniqueName: \"kubernetes.io/projected/2c012900-326d-42d5-8076-3bbbdbb1e852-kube-api-access-7hmsq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526577 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526300 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526577 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526335 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526577 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526393 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526754 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526641 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526754 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526735 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526782 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.526870 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.526827 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.527079 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.527055 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c012900-326d-42d5-8076-3bbbdbb1e852-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.528558 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.528541 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.528811 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.528794 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.546296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.546271 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c012900-326d-42d5-8076-3bbbdbb1e852-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.547310 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.547290 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmsq\" (UniqueName: \"kubernetes.io/projected/2c012900-326d-42d5-8076-3bbbdbb1e852-kube-api-access-7hmsq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x4dsn\" (UID: \"2c012900-326d-42d5-8076-3bbbdbb1e852\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.642574 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.642523 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:53.960287 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.960259 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn"] Oct 30 19:29:53.962071 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:29:53.962043 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c012900_326d_42d5_8076_3bbbdbb1e852.slice/crio-8e5d4976adec7a8f756bd25bebb82f848798b43823ba38c7df928e2212cd9ce4 WatchSource:0}: Error finding container 8e5d4976adec7a8f756bd25bebb82f848798b43823ba38c7df928e2212cd9ce4: Status 404 returned error can't find the container with id 8e5d4976adec7a8f756bd25bebb82f848798b43823ba38c7df928e2212cd9ce4 Oct 30 19:29:53.964182 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.964151 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:29:53.964255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.964221 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:29:53.964255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:53.964251 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:29:54.509307 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:29:54.509271 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:29:54.844905 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:54.844814 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" event={"ID":"2c012900-326d-42d5-8076-3bbbdbb1e852","Type":"ContainerStarted","Data":"dc914fd82568ab264f6825babae55b8dc8e5aab694fef84c00b1fe84b610e09e"} Oct 30 19:29:54.844905 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:54.844877 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" event={"ID":"2c012900-326d-42d5-8076-3bbbdbb1e852","Type":"ContainerStarted","Data":"8e5d4976adec7a8f756bd25bebb82f848798b43823ba38c7df928e2212cd9ce4"} Oct 30 19:29:54.936128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:54.932587 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" podStartSLOduration=1.93256757 podStartE2EDuration="1.93256757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:54.906450896 +0000 UTC m=+555.820055513" watchObservedRunningTime="2026-10-30 19:29:54.93256757 +0000 UTC m=+555.846172188" Oct 30 19:29:55.643438 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:55.643406 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:29:55.644974 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:55.644952 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x4dsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:55.645034 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:55.644998 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" podUID="2c012900-326d-42d5-8076-3bbbdbb1e852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:29:56.643788 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:56.643741 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x4dsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:56.644198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:56.643807 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" podUID="2c012900-326d-42d5-8076-3bbbdbb1e852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:29:57.643581 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:57.643551 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x4dsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:57.643743 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:57.643599 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" podUID="2c012900-326d-42d5-8076-3bbbdbb1e852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:29:58.643685 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:58.643652 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x4dsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:58.644068 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:58.643708 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" podUID="2c012900-326d-42d5-8076-3bbbdbb1e852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:29:59.643664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:59.643623 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x4dsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:59.643844 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:29:59.643688 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" podUID="2c012900-326d-42d5-8076-3bbbdbb1e852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:30:00.642826 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:00.642789 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x4dsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:30:00.643000 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:00.642861 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" podUID="2c012900-326d-42d5-8076-3bbbdbb1e852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:30:02.205681 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:02.205647 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:30:02.206195 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:02.206053 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:30:02.206726 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:02.206705 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" Oct 30 19:30:24.509254 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:30:24.509177 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:30:39.271447 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:39.271422 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:30:39.272855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:39.272831 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:30:39.287566 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:39.287544 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:30:39.289035 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:39.289019 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:30:54.230761 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.230728 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj"] Oct 30 19:30:54.234436 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.234417 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.253392 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.253367 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:30:54.267045 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.266998 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f913c6-serving-cert\"" Oct 30 19:30:54.267203 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.267184 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g97rg\"" Oct 30 19:30:54.267270 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.267206 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:30:54.267270 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.267230 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:30:54.270256 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.270235 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj"] Oct 30 19:30:54.328583 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.328553 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nnnx\" (UniqueName: \"kubernetes.io/projected/fd683264-cd99-440c-b137-4c8b454b492f-kube-api-access-7nnnx\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.328740 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.328613 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.328859 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.328826 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-metrics-certs\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.328940 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.328869 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd683264-cd99-440c-b137-4c8b454b492f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.429689 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.429654 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-metrics-certs\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.429689 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.429690 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd683264-cd99-440c-b137-4c8b454b492f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.429920 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.429721 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nnnx\" (UniqueName: \"kubernetes.io/projected/fd683264-cd99-440c-b137-4c8b454b492f-kube-api-access-7nnnx\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.429920 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.429747 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.430498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.430409 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd683264-cd99-440c-b137-4c8b454b492f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.432582 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.432550 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.432995 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.432972 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-metrics-certs\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.450628 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.450599 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nnnx\" (UniqueName: \"kubernetes.io/projected/fd683264-cd99-440c-b137-4c8b454b492f-kube-api-access-7nnnx\") pod \"maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.509647 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:30:54.509568 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:30:54.547845 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.547821 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:54.872875 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.872846 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj"] Oct 30 19:30:54.877317 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:54.877296 4904 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:30:55.225439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:55.225403 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" event={"ID":"fd683264-cd99-440c-b137-4c8b454b492f","Type":"ContainerStarted","Data":"4a0f8b88337aafbf43d489c5b5e18dd44a91f6632784b2dc7a54f5c7545bafec"} Oct 30 19:30:57.355519 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.355479 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq"] Oct 30 19:30:57.373783 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.373711 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.377678 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.377653 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq"] Oct 30 19:30:57.394407 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.394371 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-4b4c34-serving-cert\"" Oct 30 19:30:57.470552 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.470523 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-metrics-certs\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.470698 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.470563 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9hm\" (UniqueName: \"kubernetes.io/projected/61a7275c-1864-4f89-98c7-cef6efebe04d-kube-api-access-kd9hm\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.470698 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.470674 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.470779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.470702 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a7275c-1864-4f89-98c7-cef6efebe04d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.571827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.571792 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-metrics-certs\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.571935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.571839 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9hm\" (UniqueName: \"kubernetes.io/projected/61a7275c-1864-4f89-98c7-cef6efebe04d-kube-api-access-kd9hm\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.571935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.571883 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.571935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.571899 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a7275c-1864-4f89-98c7-cef6efebe04d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.572033 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:30:57.572016 4904 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-serving-cert: secret "maas-api-e2e-worker-w0-4b4c34-serving-cert" not found Oct 30 19:30:57.572151 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:30:57.572096 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-maas-api-tls podName:61a7275c-1864-4f89-98c7-cef6efebe04d nodeName:}" failed. No retries permitted until 2026-10-30 19:30:58.072071688 +0000 UTC m=+618.985676288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-maas-api-tls") pod "maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" (UID: "61a7275c-1864-4f89-98c7-cef6efebe04d") : secret "maas-api-e2e-worker-w0-4b4c34-serving-cert" not found Oct 30 19:30:57.572548 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.572531 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a7275c-1864-4f89-98c7-cef6efebe04d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.574121 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.574086 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-metrics-certs\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:57.592936 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:57.592914 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9hm\" (UniqueName: \"kubernetes.io/projected/61a7275c-1864-4f89-98c7-cef6efebe04d-kube-api-access-kd9hm\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:58.076811 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:58.076774 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:58.079557 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:58.079518 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:58.248630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:58.248592 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" event={"ID":"fd683264-cd99-440c-b137-4c8b454b492f","Type":"ContainerStarted","Data":"1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775"} Oct 30 19:30:58.248796 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:58.248652 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:30:58.300186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:58.300138 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" podStartSLOduration=1.640763854 podStartE2EDuration="4.300090965s" totalImagesPullingTime="2.659327111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:58.298406297 +0000 UTC m=+619.212010911" watchObservedRunningTime="2026-10-30 19:30:58.300090965 +0000 UTC m=+619.213695588" Oct 30 19:30:58.301534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:58.301495 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:58.607648 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:58.607619 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq"] Oct 30 19:30:58.608963 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:30:58.608940 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a7275c_1864_4f89_98c7_cef6efebe04d.slice/crio-9944656729f262c76be8c6ecfb3f095a94edc36574bca7900cb0dddb51c488cc WatchSource:0}: Error finding container 9944656729f262c76be8c6ecfb3f095a94edc36574bca7900cb0dddb51c488cc: Status 404 returned error can't find the container with id 9944656729f262c76be8c6ecfb3f095a94edc36574bca7900cb0dddb51c488cc Oct 30 19:30:59.257900 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:59.257864 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" event={"ID":"61a7275c-1864-4f89-98c7-cef6efebe04d","Type":"ContainerStarted","Data":"b77a67ca9c9e9447ca725ef081cf3f76cdf4864796861b64060c9b3959c1ef13"} Oct 30 19:30:59.258135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:59.257925 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" event={"ID":"61a7275c-1864-4f89-98c7-cef6efebe04d","Type":"ContainerStarted","Data":"9944656729f262c76be8c6ecfb3f095a94edc36574bca7900cb0dddb51c488cc"} Oct 30 19:30:59.258218 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:59.258142 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:30:59.301482 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:30:59.301441 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" podStartSLOduration=1.996521259 podStartE2EDuration="2.301428336s" totalImagesPullingTime="304.907077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:59.300019935 +0000 UTC m=+620.213624553" watchObservedRunningTime="2026-10-30 19:30:59.301428336 +0000 UTC m=+620.215032952" Oct 30 19:31:00.629126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.629058 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs"] Oct 30 19:31:00.632980 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.632953 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.669292 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.669262 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-d797d2-serving-cert\"" Oct 30 19:31:00.673556 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.673526 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs"] Oct 30 19:31:00.697322 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.697295 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d61769fc-6819-43b7-ab18-4b49679630b8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.697468 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.697363 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-metrics-certs\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.697468 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.697453 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnskh\" (UniqueName: \"kubernetes.io/projected/d61769fc-6819-43b7-ab18-4b49679630b8-kube-api-access-dnskh\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.697598 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.697578 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.798115 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.798066 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.798336 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.798161 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d61769fc-6819-43b7-ab18-4b49679630b8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.798336 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.798227 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-metrics-certs\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.798336 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.798266 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnskh\" (UniqueName: \"kubernetes.io/projected/d61769fc-6819-43b7-ab18-4b49679630b8-kube-api-access-dnskh\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.798691 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:31:00.798670 4904 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-serving-cert: secret "maas-api-e2e-worker-w1-d797d2-serving-cert" not found Oct 30 19:31:00.798798 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:31:00.798737 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-maas-api-tls podName:d61769fc-6819-43b7-ab18-4b49679630b8 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:01.298717159 +0000 UTC m=+622.212321774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-maas-api-tls") pod "maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" (UID: "d61769fc-6819-43b7-ab18-4b49679630b8") : secret "maas-api-e2e-worker-w1-d797d2-serving-cert" not found Oct 30 19:31:00.799506 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.799484 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d61769fc-6819-43b7-ab18-4b49679630b8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.802211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.802185 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-metrics-certs\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:00.822846 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:00.822824 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnskh\" (UniqueName: \"kubernetes.io/projected/d61769fc-6819-43b7-ab18-4b49679630b8-kube-api-access-dnskh\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:01.303818 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:01.303780 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:01.306396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:01.306370 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:01.566376 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:01.566275 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:01.913158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:01.913126 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs"] Oct 30 19:31:01.914923 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:31:01.914885 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61769fc_6819_43b7_ab18_4b49679630b8.slice/crio-7263037b6485cc0c2fb20ed033a8b16c5af5e40fb06f9c376bab2a0dec0cad74 WatchSource:0}: Error finding container 7263037b6485cc0c2fb20ed033a8b16c5af5e40fb06f9c376bab2a0dec0cad74: Status 404 returned error can't find the container with id 7263037b6485cc0c2fb20ed033a8b16c5af5e40fb06f9c376bab2a0dec0cad74 Oct 30 19:31:02.278824 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:02.278761 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" event={"ID":"d61769fc-6819-43b7-ab18-4b49679630b8","Type":"ContainerStarted","Data":"7263037b6485cc0c2fb20ed033a8b16c5af5e40fb06f9c376bab2a0dec0cad74"} Oct 30 19:31:03.286034 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:03.286003 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" event={"ID":"d61769fc-6819-43b7-ab18-4b49679630b8","Type":"ContainerStarted","Data":"8c9a2f30d6f3b4754978331e6e79a5b3abec0505921d7fd30d3ff5f60b976a7e"} Oct 30 19:31:03.286465 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:03.286057 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:03.329576 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:03.329530 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" podStartSLOduration=2.974662455 podStartE2EDuration="3.329514722s" totalImagesPullingTime="354.852267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:03.328461088 +0000 UTC m=+624.242065704" watchObservedRunningTime="2026-10-30 19:31:03.329514722 +0000 UTC m=+624.243119340" Oct 30 19:31:04.263763 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:04.263732 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:31:05.269874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:05.269846 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:31:09.297384 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.297351 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:31:09.824433 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.824397 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f"] Oct 30 19:31:09.828816 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.828794 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.847458 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.847431 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f"] Oct 30 19:31:09.856712 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.856669 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef9a5a\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.856874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.856855 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef9a5a\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.858934 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.858915 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef9a5a\"/\"simulator-w6-ef9a5a-kserve-self-signed-certs\"" Oct 30 19:31:09.859052 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.858937 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef9a5a\"/\"default-dockercfg-wtrg6\"" Oct 30 19:31:09.893956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.893928 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-dshm\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.894056 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.893974 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-model-cache\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.894056 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.893993 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-tmp-dir\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.894179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.894061 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqggl\" (UniqueName: \"kubernetes.io/projected/c37ac3f1-7271-41db-8d39-9194a2820225-kube-api-access-vqggl\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.894179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.894096 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c37ac3f1-7271-41db-8d39-9194a2820225-tls-certs\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.894299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.894277 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-home\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.995912 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.995874 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-dshm\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.996089 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.995935 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-model-cache\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.996089 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.995970 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-tmp-dir\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.996089 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.995997 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqggl\" (UniqueName: \"kubernetes.io/projected/c37ac3f1-7271-41db-8d39-9194a2820225-kube-api-access-vqggl\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.996089 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.996036 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c37ac3f1-7271-41db-8d39-9194a2820225-tls-certs\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.996344 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.996118 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-home\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.996402 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.996383 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-model-cache\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.996515 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.996490 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-tmp-dir\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.996590 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.996542 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-home\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.998431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.998405 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-dshm\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:09.998865 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:09.998849 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c37ac3f1-7271-41db-8d39-9194a2820225-tls-certs\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:10.016746 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:10.016724 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqggl\" (UniqueName: \"kubernetes.io/projected/c37ac3f1-7271-41db-8d39-9194a2820225-kube-api-access-vqggl\") pod \"simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:10.144620 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:10.144541 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:10.455010 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:10.454980 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f"] Oct 30 19:31:10.456201 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:31:10.456170 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc37ac3f1_7271_41db_8d39_9194a2820225.slice/crio-db582b4f18d1a252b2564ed3a302929375039270ed3cbbcfee38163f9d6b75bd WatchSource:0}: Error finding container db582b4f18d1a252b2564ed3a302929375039270ed3cbbcfee38163f9d6b75bd: Status 404 returned error can't find the container with id db582b4f18d1a252b2564ed3a302929375039270ed3cbbcfee38163f9d6b75bd Oct 30 19:31:11.346156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:11.346083 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" event={"ID":"c37ac3f1-7271-41db-8d39-9194a2820225","Type":"ContainerStarted","Data":"db582b4f18d1a252b2564ed3a302929375039270ed3cbbcfee38163f9d6b75bd"} Oct 30 19:31:12.359252 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:12.359210 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" event={"ID":"c37ac3f1-7271-41db-8d39-9194a2820225","Type":"ContainerStarted","Data":"fc563fea9cb8854c3b04b03b49025d3b65468da06402a3a9dc2c2981a5424f56"} Oct 30 19:31:12.400936 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:12.400894 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podStartSLOduration=2.174842771 podStartE2EDuration="3.400876342s" totalImagesPullingTime="1.226033571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:12.400341888 +0000 UTC m=+633.313946507" watchObservedRunningTime="2026-10-30 19:31:12.400876342 +0000 UTC m=+633.314480959" Oct 30 19:31:13.144891 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:13.144860 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:20.145682 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:20.145643 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:20.158874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:20.158850 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:20.425270 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:20.425244 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:31:21.082574 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.082528 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5"] Oct 30 19:31:21.090824 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.090764 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.107118 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.107080 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5"] Oct 30 19:31:21.123144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.117959 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-544fc8\"/\"simulator-w5-544fc8-kserve-self-signed-certs\"" Oct 30 19:31:21.123144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.117968 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-544fc8\"/\"default-dockercfg-g5jtg\"" Oct 30 19:31:21.123144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.121729 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-544fc8\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.123144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.121895 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-544fc8\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.230400 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.230372 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-model-cache\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.230400 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.230403 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tmp-dir\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.230957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.230436 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tls-certs\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.230957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.230483 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjj7g\" (UniqueName: \"kubernetes.io/projected/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-kube-api-access-bjj7g\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.230957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.230604 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-home\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.230957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.230722 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-dshm\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.331549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.331520 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tls-certs\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.331724 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.331556 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjj7g\" (UniqueName: \"kubernetes.io/projected/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-kube-api-access-bjj7g\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.331724 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.331602 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-home\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.331724 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.331668 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-dshm\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.331888 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.331783 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-model-cache\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.331888 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.331809 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tmp-dir\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.332161 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.332129 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-home\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.332264 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.332195 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-model-cache\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.332329 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.332278 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tmp-dir\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.334130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.334063 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-dshm\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.334407 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.334388 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tls-certs\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.353483 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.353454 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjj7g\" (UniqueName: \"kubernetes.io/projected/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-kube-api-access-bjj7g\") pod \"simulator-w5-544fc8-kserve-65bdc55656-nqqk5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.407993 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.407962 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:21.735208 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:21.735185 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5"] Oct 30 19:31:21.736658 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:31:21.736632 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63ce9a7b_eec0_40d9_b0f7_829f261be3e5.slice/crio-a80d5d110651e38ec233ed76564eb161e9321e0ecd24c3ceb86d8c2a9b8fb8be WatchSource:0}: Error finding container a80d5d110651e38ec233ed76564eb161e9321e0ecd24c3ceb86d8c2a9b8fb8be: Status 404 returned error can't find the container with id a80d5d110651e38ec233ed76564eb161e9321e0ecd24c3ceb86d8c2a9b8fb8be Oct 30 19:31:22.430726 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:22.430687 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" event={"ID":"63ce9a7b-eec0-40d9-b0f7-829f261be3e5","Type":"ContainerStarted","Data":"775f2663ece223f437cd77fa72248ae21a000fdc2545be0566ff660d51b2076e"} Oct 30 19:31:22.430726 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:22.430727 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" event={"ID":"63ce9a7b-eec0-40d9-b0f7-829f261be3e5","Type":"ContainerStarted","Data":"a80d5d110651e38ec233ed76564eb161e9321e0ecd24c3ceb86d8c2a9b8fb8be"} Oct 30 19:31:22.477272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:22.477218 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podStartSLOduration=1.477204174 podStartE2EDuration="1.477204174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:22.47532647 +0000 UTC m=+643.388931086" watchObservedRunningTime="2026-10-30 19:31:22.477204174 +0000 UTC m=+643.390808825" Oct 30 19:31:23.409003 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:23.408970 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:24.515371 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:31:24.515331 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:31:26.002679 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.002650 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5"] Oct 30 19:31:26.009540 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.009515 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.025317 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.025288 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5"] Oct 30 19:31:26.035074 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.035050 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f913c6\"/\"default-dockercfg-bcjnr\"" Oct 30 19:31:26.035242 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.035169 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f913c6\"/\"simulator-w4-f913c6-kserve-self-signed-certs\"" Oct 30 19:31:26.077467 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.077447 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f913c6\"/\"openshift-service-ca.crt\"" Oct 30 19:31:26.077610 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.077591 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f913c6\"/\"kube-root-ca.crt\"" Oct 30 19:31:26.080087 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.080069 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51692e8-fa0c-4042-8255-be65097e79a9-tls-certs\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.080177 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.080140 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-home\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.080271 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.080254 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-dshm\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.080333 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.080286 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-tmp-dir\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.080385 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.080331 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgrr\" (UniqueName: \"kubernetes.io/projected/e51692e8-fa0c-4042-8255-be65097e79a9-kube-api-access-5fgrr\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.080440 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.080402 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-model-cache\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.181948 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.181903 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51692e8-fa0c-4042-8255-be65097e79a9-tls-certs\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.182143 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.181953 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-home\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.182143 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.182009 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-dshm\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.182143 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.182031 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-tmp-dir\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.182143 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.182084 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgrr\" (UniqueName: \"kubernetes.io/projected/e51692e8-fa0c-4042-8255-be65097e79a9-kube-api-access-5fgrr\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.182361 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.182149 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-model-cache\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.182581 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.182461 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-home\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.182581 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.182516 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-model-cache\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.182744 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.182655 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-tmp-dir\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.184834 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.184807 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51692e8-fa0c-4042-8255-be65097e79a9-tls-certs\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.184921 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.184904 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-dshm\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.203004 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.202977 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgrr\" (UniqueName: \"kubernetes.io/projected/e51692e8-fa0c-4042-8255-be65097e79a9-kube-api-access-5fgrr\") pod \"simulator-w4-f913c6-kserve-5fd98c474c-dhmw5\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.326559 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.326473 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:26.634871 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:26.634847 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5"] Oct 30 19:31:26.636985 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:31:26.636952 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode51692e8_fa0c_4042_8255_be65097e79a9.slice/crio-1255cc6397423d014fcf3d846275236a3b9448e2c82433d35fad0232d511c4c6 WatchSource:0}: Error finding container 1255cc6397423d014fcf3d846275236a3b9448e2c82433d35fad0232d511c4c6: Status 404 returned error can't find the container with id 1255cc6397423d014fcf3d846275236a3b9448e2c82433d35fad0232d511c4c6 Oct 30 19:31:27.466928 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:27.466888 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" event={"ID":"e51692e8-fa0c-4042-8255-be65097e79a9","Type":"ContainerStarted","Data":"98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d"} Oct 30 19:31:27.466928 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:27.466932 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" event={"ID":"e51692e8-fa0c-4042-8255-be65097e79a9","Type":"ContainerStarted","Data":"1255cc6397423d014fcf3d846275236a3b9448e2c82433d35fad0232d511c4c6"} Oct 30 19:31:27.511925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:27.511863 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podStartSLOduration=2.511848507 podStartE2EDuration="2.511848507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:27.510242776 +0000 UTC m=+648.423847393" watchObservedRunningTime="2026-10-30 19:31:27.511848507 +0000 UTC m=+648.425453124" Oct 30 19:31:28.327191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:28.327161 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:31.408507 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:31.408461 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:31.427081 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:31.427056 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:31.512240 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:31.512213 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:31:36.172466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.172429 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m"] Oct 30 19:31:36.182374 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.182348 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.194861 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.194835 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m"] Oct 30 19:31:36.202020 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.201996 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d797d2\"/\"openshift-service-ca.crt\"" Oct 30 19:31:36.204215 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.204191 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d797d2\"/\"kube-root-ca.crt\"" Oct 30 19:31:36.204345 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.204305 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d797d2\"/\"default-dockercfg-mlm87\"" Oct 30 19:31:36.211581 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.211558 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d797d2\"/\"simulator-w1-d797d2-kserve-self-signed-certs\"" Oct 30 19:31:36.292244 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.292210 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-dshm\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.292434 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.292255 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4k2\" (UniqueName: \"kubernetes.io/projected/a8ea566e-8d44-469a-bd0b-2b2932b16679-kube-api-access-rg4k2\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.292434 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.292374 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-home\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.292548 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.292450 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-tmp-dir\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.292614 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.292595 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8ea566e-8d44-469a-bd0b-2b2932b16679-tls-certs\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.292764 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.292741 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-model-cache\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.326648 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.326618 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:36.345449 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.345407 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:36.393394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.393355 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-model-cache\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.393568 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.393428 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-dshm\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.393568 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.393460 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4k2\" (UniqueName: \"kubernetes.io/projected/a8ea566e-8d44-469a-bd0b-2b2932b16679-kube-api-access-rg4k2\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.393568 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.393492 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-home\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.393568 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.393540 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-tmp-dir\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.393859 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.393602 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8ea566e-8d44-469a-bd0b-2b2932b16679-tls-certs\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.393859 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.393786 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-model-cache\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.394142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.394092 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-home\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.394247 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.394150 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-tmp-dir\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.396368 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.396341 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-dshm\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.401502 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.397550 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8ea566e-8d44-469a-bd0b-2b2932b16679-tls-certs\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.429168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.428806 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4k2\" (UniqueName: \"kubernetes.io/projected/a8ea566e-8d44-469a-bd0b-2b2932b16679-kube-api-access-rg4k2\") pod \"simulator-w1-d797d2-kserve-5b86b9f589-bjn2m\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.502165 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.502132 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:36.556410 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.553786 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:31:36.812671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:36.812639 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m"] Oct 30 19:31:36.813493 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:31:36.813466 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8ea566e_8d44_469a_bd0b_2b2932b16679.slice/crio-3d389157625ddc99b92a063c5dac43b29f4b7eb8aaf27f302fa7007fe4203d89 WatchSource:0}: Error finding container 3d389157625ddc99b92a063c5dac43b29f4b7eb8aaf27f302fa7007fe4203d89: Status 404 returned error can't find the container with id 3d389157625ddc99b92a063c5dac43b29f4b7eb8aaf27f302fa7007fe4203d89 Oct 30 19:31:37.542959 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:37.542915 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" event={"ID":"a8ea566e-8d44-469a-bd0b-2b2932b16679","Type":"ContainerStarted","Data":"57965e0a4d8195629dee879f79f939a8e73f7b50dab57ed34258f6a52c387c4a"} Oct 30 19:31:37.542959 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:37.542972 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" event={"ID":"a8ea566e-8d44-469a-bd0b-2b2932b16679","Type":"ContainerStarted","Data":"3d389157625ddc99b92a063c5dac43b29f4b7eb8aaf27f302fa7007fe4203d89"} Oct 30 19:31:37.587388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:37.587340 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podStartSLOduration=1.587327858 podStartE2EDuration="1.587327858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:37.584893905 +0000 UTC m=+658.498498521" watchObservedRunningTime="2026-10-30 19:31:37.587327858 +0000 UTC m=+658.500932475" Oct 30 19:31:38.503187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:38.503154 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:46.503549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:46.503514 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:46.527684 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:46.527655 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:46.631562 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:46.631518 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:31:47.278073 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.278026 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj"] Oct 30 19:31:47.284529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.284500 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj"] Oct 30 19:31:47.284702 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.284593 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.303086 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.303059 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef9a5a\"/\"premium-w6-ef9a5a-kserve-self-signed-certs\"" Oct 30 19:31:47.307671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.307644 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-home\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.307797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.307754 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4cj\" (UniqueName: \"kubernetes.io/projected/87aab154-3c67-4749-a67d-365be3e3437d-kube-api-access-xk4cj\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.307846 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.307818 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-model-cache\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.307892 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.307863 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87aab154-3c67-4749-a67d-365be3e3437d-tls-certs\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.307926 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.307901 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-tmp-dir\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.307926 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.307918 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-dshm\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.409178 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.409130 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87aab154-3c67-4749-a67d-365be3e3437d-tls-certs\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.409364 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.409200 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-tmp-dir\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.409364 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.409227 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-dshm\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.409364 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.409260 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-home\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.409364 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.409352 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4cj\" (UniqueName: \"kubernetes.io/projected/87aab154-3c67-4749-a67d-365be3e3437d-kube-api-access-xk4cj\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.409582 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.409429 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-model-cache\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.409735 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.409711 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-tmp-dir\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.409831 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.409806 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-model-cache\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.410070 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.410043 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-home\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.411775 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.411750 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-dshm\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.412289 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.412260 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87aab154-3c67-4749-a67d-365be3e3437d-tls-certs\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.431519 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.431494 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4cj\" (UniqueName: \"kubernetes.io/projected/87aab154-3c67-4749-a67d-365be3e3437d-kube-api-access-xk4cj\") pod \"premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.598951 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.598854 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:47.922080 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:47.922051 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj"] Oct 30 19:31:47.923962 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:31:47.923932 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87aab154_3c67_4749_a67d_365be3e3437d.slice/crio-118094aeb52feef6a8cf1ba811fbac415d918b6a84c8e6c86435422e056fdbd6 WatchSource:0}: Error finding container 118094aeb52feef6a8cf1ba811fbac415d918b6a84c8e6c86435422e056fdbd6: Status 404 returned error can't find the container with id 118094aeb52feef6a8cf1ba811fbac415d918b6a84c8e6c86435422e056fdbd6 Oct 30 19:31:48.631678 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:48.631640 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" event={"ID":"87aab154-3c67-4749-a67d-365be3e3437d","Type":"ContainerStarted","Data":"375dc63220f78823ba5d487c692898ecdbc0f78deecbe9159b9ded5cff160a0c"} Oct 30 19:31:48.632068 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:48.631691 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" event={"ID":"87aab154-3c67-4749-a67d-365be3e3437d","Type":"ContainerStarted","Data":"118094aeb52feef6a8cf1ba811fbac415d918b6a84c8e6c86435422e056fdbd6"} Oct 30 19:31:48.674538 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:48.674496 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podStartSLOduration=1.674481647 podStartE2EDuration="1.674481647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:48.672359674 +0000 UTC m=+669.585964291" watchObservedRunningTime="2026-10-30 19:31:48.674481647 +0000 UTC m=+669.588086263" Oct 30 19:31:49.599407 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:49.599330 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:54.509057 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:31:54.509017 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:31:56.690523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.690489 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2"] Oct 30 19:31:56.694569 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.694544 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.715494 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.715463 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2"] Oct 30 19:31:56.724578 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.724556 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f913c6\"/\"premium-w4-f913c6-kserve-self-signed-certs\"" Oct 30 19:31:56.798563 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.798536 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-model-cache\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.798697 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.798592 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-tmp-dir\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.798697 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.798610 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-home\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.798776 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.798759 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmh5\" (UniqueName: \"kubernetes.io/projected/b26d1d6e-fb62-4e3c-9913-2944866d3508-kube-api-access-qgmh5\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.798826 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.798809 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-dshm\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.798990 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.798965 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b26d1d6e-fb62-4e3c-9913-2944866d3508-tls-certs\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.901704 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.901660 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-model-cache\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.901950 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.901748 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-tmp-dir\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.901950 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.901778 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-home\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.901950 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.901819 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmh5\" (UniqueName: \"kubernetes.io/projected/b26d1d6e-fb62-4e3c-9913-2944866d3508-kube-api-access-qgmh5\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.901950 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.901858 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-dshm\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.901950 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.901940 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b26d1d6e-fb62-4e3c-9913-2944866d3508-tls-certs\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.902299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.902139 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-model-cache\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.902501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.902477 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-home\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.902768 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.902745 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-tmp-dir\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.907617 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.907591 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-dshm\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.910764 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.908870 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b26d1d6e-fb62-4e3c-9913-2944866d3508-tls-certs\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:56.926157 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:56.924852 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmh5\" (UniqueName: \"kubernetes.io/projected/b26d1d6e-fb62-4e3c-9913-2944866d3508-kube-api-access-qgmh5\") pod \"premium-w4-f913c6-kserve-96d7bd8b6-7cwj2\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:57.012938 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:57.012871 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:31:57.321044 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:57.320971 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2"] Oct 30 19:31:57.324642 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:31:57.324608 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb26d1d6e_fb62_4e3c_9913_2944866d3508.slice/crio-5ff0603c05bc857ff1143f59bd29ccb288241351e013e7d658b67ab60a0a9814 WatchSource:0}: Error finding container 5ff0603c05bc857ff1143f59bd29ccb288241351e013e7d658b67ab60a0a9814: Status 404 returned error can't find the container with id 5ff0603c05bc857ff1143f59bd29ccb288241351e013e7d658b67ab60a0a9814 Oct 30 19:31:57.599797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:57.599718 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:57.613207 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:57.613180 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:57.693685 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:57.693650 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" event={"ID":"b26d1d6e-fb62-4e3c-9913-2944866d3508","Type":"ContainerStarted","Data":"08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8"} Oct 30 19:31:57.694180 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:57.693695 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" event={"ID":"b26d1d6e-fb62-4e3c-9913-2944866d3508","Type":"ContainerStarted","Data":"5ff0603c05bc857ff1143f59bd29ccb288241351e013e7d658b67ab60a0a9814"} Oct 30 19:31:57.705092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:57.705068 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:31:57.740274 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:57.740235 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podStartSLOduration=1.740221484 podStartE2EDuration="1.740221484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:57.738717205 +0000 UTC m=+678.652321822" watchObservedRunningTime="2026-10-30 19:31:57.740221484 +0000 UTC m=+678.653826115" Oct 30 19:31:58.013543 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:31:58.013509 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:32:07.013531 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:07.013489 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:32:07.026377 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:07.026344 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:32:07.780142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:07.780088 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:32:20.344364 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.344334 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4"] Oct 30 19:32:20.349606 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.349580 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.367573 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.367537 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4"] Oct 30 19:32:20.376378 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.376349 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd23d1\"/\"openshift-service-ca.crt\"" Oct 30 19:32:20.376560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.376360 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd23d1\"/\"kube-root-ca.crt\"" Oct 30 19:32:20.379347 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.379327 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd23d1\"/\"default-dockercfg-tc9vt\"" Oct 30 19:32:20.379421 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.379350 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd23d1\"/\"premium-w3-dd23d1-kserve-self-signed-certs\"" Oct 30 19:32:20.448796 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.448759 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8pp\" (UniqueName: \"kubernetes.io/projected/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-kube-api-access-rj8pp\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.448945 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.448838 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-dshm\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.449002 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.448947 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-home\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.449061 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.449044 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tls-certs\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.449133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.449076 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tmp-dir\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.449188 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.449132 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-model-cache\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.549959 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.549925 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tls-certs\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.550244 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.550226 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tmp-dir\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.550402 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.550389 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-model-cache\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.550548 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.550535 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8pp\" (UniqueName: \"kubernetes.io/projected/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-kube-api-access-rj8pp\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.550693 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.550681 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-dshm\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.550810 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.550797 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-home\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.551365 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.551339 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-home\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.554554 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.554526 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-model-cache\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.554818 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.554796 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tmp-dir\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.555773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.555753 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tls-certs\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.558401 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.558377 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-dshm\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.577608 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.577583 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8pp\" (UniqueName: \"kubernetes.io/projected/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-kube-api-access-rj8pp\") pod \"premium-w3-dd23d1-kserve-7d7794cc95-6hmt4\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.664973 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.664898 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:20.969884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:20.969857 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4"] Oct 30 19:32:20.971504 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:32:20.971478 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc111f7d7_b7f6_4ed2_8087_e4fe9d307eed.slice/crio-5f6a98be845c6a3d4686d6cd8dbc5326eb8fe7b6a308ef6333f29e83abcd53a1 WatchSource:0}: Error finding container 5f6a98be845c6a3d4686d6cd8dbc5326eb8fe7b6a308ef6333f29e83abcd53a1: Status 404 returned error can't find the container with id 5f6a98be845c6a3d4686d6cd8dbc5326eb8fe7b6a308ef6333f29e83abcd53a1 Oct 30 19:32:21.864983 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:21.864945 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" event={"ID":"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed","Type":"ContainerStarted","Data":"798dd70f34b376b441f9e331c2d325efaedc72c6fd8c00e2a4a38e662d5b0d8a"} Oct 30 19:32:21.865410 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:21.864997 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" event={"ID":"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed","Type":"ContainerStarted","Data":"5f6a98be845c6a3d4686d6cd8dbc5326eb8fe7b6a308ef6333f29e83abcd53a1"} Oct 30 19:32:21.910254 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:21.910204 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podStartSLOduration=1.910191451 podStartE2EDuration="1.910191451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:21.909337047 +0000 UTC m=+702.822941665" watchObservedRunningTime="2026-10-30 19:32:21.910191451 +0000 UTC m=+702.823796069" Oct 30 19:32:22.665048 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:22.665019 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:24.511282 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:32:24.511241 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:32:30.665307 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:30.665275 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:30.679009 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:30.678982 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:30.941315 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:32:30.941289 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:32:54.513287 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:32:54.513244 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:33:24.511205 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:33:24.511123 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:33:30.263663 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.263627 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd"] Oct 30 19:33:30.267737 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.267714 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.286861 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.286836 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd"] Oct 30 19:33:30.296866 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.296844 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd23d1-openshift-default-dockercfg-qvtg4\"" Oct 30 19:33:30.366714 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.366679 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.366867 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.366741 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.366908 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.366873 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.366943 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.366917 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zck4\" (UniqueName: \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-kube-api-access-4zck4\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.367065 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.367041 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.367132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.367090 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.367172 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.367132 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.367226 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.367210 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.367267 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.367255 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.468619 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.468581 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.468619 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.468624 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.468858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.468666 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.468858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.468689 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zck4\" (UniqueName: \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-kube-api-access-4zck4\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.468858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.468726 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.468858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.468749 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.468858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.468766 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.468858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.468795 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.468858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.468827 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.469294 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.469197 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.469294 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.469260 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.469419 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.469299 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.469509 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.469487 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.469718 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.469671 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.471369 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.471342 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.471613 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.471594 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.490967 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.490942 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.492116 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.492081 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zck4\" (UniqueName: \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-kube-api-access-4zck4\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.581480 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.581413 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:30.909043 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.909010 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd"] Oct 30 19:33:30.910717 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:33:30.910682 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa2d4f9_8b9e_4915_8753_62f185d5a7ef.slice/crio-cbbdcb1c57c92c82de98885bd26db13d157cfa16f8dcfb1e2d4a85fcda888837 WatchSource:0}: Error finding container cbbdcb1c57c92c82de98885bd26db13d157cfa16f8dcfb1e2d4a85fcda888837: Status 404 returned error can't find the container with id cbbdcb1c57c92c82de98885bd26db13d157cfa16f8dcfb1e2d4a85fcda888837 Oct 30 19:33:30.913643 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.913608 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:33:30.913729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.913672 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:33:30.913729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:30.913698 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:33:31.371567 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:31.371523 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" event={"ID":"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef","Type":"ContainerStarted","Data":"c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84"} Oct 30 19:33:31.372189 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:31.371580 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" event={"ID":"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef","Type":"ContainerStarted","Data":"cbbdcb1c57c92c82de98885bd26db13d157cfa16f8dcfb1e2d4a85fcda888837"} Oct 30 19:33:31.454447 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:31.454381 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" podStartSLOduration=1.454360663 podStartE2EDuration="1.454360663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:31.451562006 +0000 UTC m=+772.365166623" watchObservedRunningTime="2026-10-30 19:33:31.454360663 +0000 UTC m=+772.367965275" Oct 30 19:33:31.584129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:31.582636 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:31.593937 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:31.590124 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:33:31.593937 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:31.590176 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:33:32.583613 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:32.583573 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:33:32.584163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:32.583638 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:33:33.582845 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:33.582811 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:33:33.583156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:33.583128 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:33:34.587447 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:34.587413 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:33:34.588015 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:34.587985 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:33:35.582315 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:35.582277 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:33:35.582487 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:35.582347 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:33:36.582678 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:36.582645 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:33:36.583208 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:36.582705 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:33:37.582384 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:37.582356 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:33:37.582539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:37.582403 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:33:38.587653 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:38.587618 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:38.588161 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:38.588149 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:38.588866 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:38.588842 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:33:44.594955 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.594920 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx"] Oct 30 19:33:44.600593 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.600576 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.633438 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.631375 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ef9a5a-openshift-default-dockercfg-dtqfz\"" Oct 30 19:33:44.637561 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.637532 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx"] Oct 30 19:33:44.660919 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.660888 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cf6p\" (UniqueName: \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-kube-api-access-9cf6p\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.661060 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.660965 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.661205 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.661075 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.661205 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.661193 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.661362 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.661332 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.661421 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.661385 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.661506 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.661483 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.661567 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.661545 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.661623 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.661573 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763087 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763297 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763140 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763297 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763168 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763297 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763209 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cf6p\" (UniqueName: \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-kube-api-access-9cf6p\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763297 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763255 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763297 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763287 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763576 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763344 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763576 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763409 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763576 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763449 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763790 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763762 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763907 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763881 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763904 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763919 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.763984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.763967 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.765965 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.765939 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.766132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.766089 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.784329 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.784306 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.784568 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.784551 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cf6p\" (UniqueName: \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-kube-api-access-9cf6p\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:44.914570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:44.914502 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:45.247945 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:45.247915 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx"] Oct 30 19:33:45.249304 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:33:45.249276 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46da1cc6_92fa_47a0_91a8_eaf26cd86427.slice/crio-a86c13de3021f60797b039fe720f3e2201d4af948d5f77d1d0bccf2f5c010bf9 WatchSource:0}: Error finding container a86c13de3021f60797b039fe720f3e2201d4af948d5f77d1d0bccf2f5c010bf9: Status 404 returned error can't find the container with id a86c13de3021f60797b039fe720f3e2201d4af948d5f77d1d0bccf2f5c010bf9 Oct 30 19:33:45.251671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:45.251633 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:33:45.251783 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:45.251700 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:33:45.251783 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:45.251732 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:33:45.523424 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:45.523340 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" event={"ID":"46da1cc6-92fa-47a0-91a8-eaf26cd86427","Type":"ContainerStarted","Data":"a9b4a0c414648ee590b2ca613e85cd137ab1bda437bcd26c04eb0c1ef4b9fba1"} Oct 30 19:33:45.523424 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:45.523411 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" event={"ID":"46da1cc6-92fa-47a0-91a8-eaf26cd86427","Type":"ContainerStarted","Data":"a86c13de3021f60797b039fe720f3e2201d4af948d5f77d1d0bccf2f5c010bf9"} Oct 30 19:33:45.571730 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:45.571680 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" podStartSLOduration=1.571664228 podStartE2EDuration="1.571664228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:45.567597642 +0000 UTC m=+786.481202260" watchObservedRunningTime="2026-10-30 19:33:45.571664228 +0000 UTC m=+786.485268847" Oct 30 19:33:45.939189 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:45.916200 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:46.939882 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:46.939845 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:33:46.940334 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:46.939924 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:33:47.918494 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:47.918455 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:33:47.918796 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:47.918767 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:33:48.924128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:48.922449 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:33:48.924128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:48.922508 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:33:49.938126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:49.937460 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:33:49.938126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:49.937518 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:33:50.916555 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:50.916518 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:33:50.916829 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:50.916797 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:33:52.765611 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:52.765577 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:52.766137 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:52.765980 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:52.766867 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:52.766845 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:33:54.003133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.003078 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6"] Oct 30 19:33:54.007400 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.007379 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.029912 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.029886 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6"] Oct 30 19:33:54.119777 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.119746 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7vxd\" (UniqueName: \"kubernetes.io/projected/8da395a9-9be0-46ba-b397-ee54354bf861-kube-api-access-n7vxd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.119924 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.119784 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.119924 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.119810 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.119924 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.119917 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da395a9-9be0-46ba-b397-ee54354bf861-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.120025 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.119964 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.120092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.120064 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da395a9-9be0-46ba-b397-ee54354bf861-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.120168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.120137 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.120283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.120259 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.120318 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.120298 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da395a9-9be0-46ba-b397-ee54354bf861-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.221911 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.221858 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.222162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.221935 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da395a9-9be0-46ba-b397-ee54354bf861-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.222162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.221969 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.222162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.222021 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.222162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.222046 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da395a9-9be0-46ba-b397-ee54354bf861-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.222162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.222091 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7vxd\" (UniqueName: \"kubernetes.io/projected/8da395a9-9be0-46ba-b397-ee54354bf861-kube-api-access-n7vxd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.222162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.222144 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.222716 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.222169 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.222716 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.222237 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da395a9-9be0-46ba-b397-ee54354bf861-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.222716 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.222309 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.223002 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.222981 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.223070 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.223016 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da395a9-9be0-46ba-b397-ee54354bf861-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.223214 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.223196 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.223400 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.223379 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.225410 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.225379 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da395a9-9be0-46ba-b397-ee54354bf861-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.225683 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.225666 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da395a9-9be0-46ba-b397-ee54354bf861-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.243836 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.243809 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da395a9-9be0-46ba-b397-ee54354bf861-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.243836 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.243820 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7vxd\" (UniqueName: \"kubernetes.io/projected/8da395a9-9be0-46ba-b397-ee54354bf861-kube-api-access-n7vxd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8k4b6\" (UID: \"8da395a9-9be0-46ba-b397-ee54354bf861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.322885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.322808 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:54.520147 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:33:54.518734 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:33:54.662437 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.662408 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6"] Oct 30 19:33:54.665596 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.665564 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:33:54.665707 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.665642 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:33:54.665707 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:54.665685 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:33:55.648753 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:55.648712 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" event={"ID":"8da395a9-9be0-46ba-b397-ee54354bf861","Type":"ContainerStarted","Data":"5ee6435425f10d16839f150b0328bd4240d76602ce44ac28631ea056a1ddb40d"} Oct 30 19:33:55.649349 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:55.648780 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" event={"ID":"8da395a9-9be0-46ba-b397-ee54354bf861","Type":"ContainerStarted","Data":"cd77c1878bb5dcdf5093ed6649978ec18c5f039b4181844dac836d7b94ebf13f"} Oct 30 19:33:55.700435 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:55.700364 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" podStartSLOduration=2.700346432 podStartE2EDuration="2.700346432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:55.697995874 +0000 UTC m=+796.611600493" watchObservedRunningTime="2026-10-30 19:33:55.700346432 +0000 UTC m=+796.613951050" Oct 30 19:33:56.325127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:56.324452 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:33:56.329132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:56.328487 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8k4b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:33:56.329132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:56.328542 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" podUID="8da395a9-9be0-46ba-b397-ee54354bf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:33:57.330126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:57.328452 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8k4b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:33:57.330126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:57.328517 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" podUID="8da395a9-9be0-46ba-b397-ee54354bf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:33:58.324125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.323407 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8k4b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:33:58.324125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.323466 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" podUID="8da395a9-9be0-46ba-b397-ee54354bf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:33:58.754332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.754297 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw"] Oct 30 19:33:58.759235 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.759210 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.777131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.777085 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw"] Oct 30 19:33:58.780342 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.780308 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tls-certs\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.780490 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.780371 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-model-cache\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.780490 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.780397 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcnl4\" (UniqueName: \"kubernetes.io/projected/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-kube-api-access-dcnl4\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.780620 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.780490 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-home\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.780620 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.780531 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-dshm\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.780729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.780633 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tmp-dir\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.786394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.786370 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3735b2\"/\"default-dockercfg-w7mcc\"" Oct 30 19:33:58.786514 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.786458 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3735b2\"/\"distinct-w2-3735b2-kserve-self-signed-certs\"" Oct 30 19:33:58.788264 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.788243 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3735b2\"/\"openshift-service-ca.crt\"" Oct 30 19:33:58.788371 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.788313 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3735b2\"/\"kube-root-ca.crt\"" Oct 30 19:33:58.881421 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.881378 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tmp-dir\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.881612 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.881430 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tls-certs\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.881612 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.881471 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-model-cache\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.881612 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.881493 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcnl4\" (UniqueName: \"kubernetes.io/projected/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-kube-api-access-dcnl4\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.881612 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.881570 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-home\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.881612 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.881607 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-dshm\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.881907 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.881887 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tmp-dir\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.882007 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.881981 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-home\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.882636 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.882520 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-model-cache\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.884776 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.884755 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-dshm\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.885194 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.885169 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tls-certs\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:58.903712 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:58.903693 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcnl4\" (UniqueName: \"kubernetes.io/projected/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-kube-api-access-dcnl4\") pod \"distinct-w2-3735b2-kserve-6f6f588d9c-764rw\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:59.079720 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.079432 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:33:59.345217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.339440 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8k4b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:33:59.345217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.339496 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" podUID="8da395a9-9be0-46ba-b397-ee54354bf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:33:59.444621 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.444592 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw"] Oct 30 19:33:59.447995 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:33:59.447954 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c3b2f6a_9984_4f4f_9874_24a3bf955a27.slice/crio-a64f8f74a8537a8a41c23a1eb5e038452d8830aad17c0df59cd38fe345c874e0 WatchSource:0}: Error finding container a64f8f74a8537a8a41c23a1eb5e038452d8830aad17c0df59cd38fe345c874e0: Status 404 returned error can't find the container with id a64f8f74a8537a8a41c23a1eb5e038452d8830aad17c0df59cd38fe345c874e0 Oct 30 19:33:59.677938 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.677907 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" event={"ID":"0c3b2f6a-9984-4f4f-9874-24a3bf955a27","Type":"ContainerStarted","Data":"b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599"} Oct 30 19:33:59.678135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.677957 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" event={"ID":"0c3b2f6a-9984-4f4f-9874-24a3bf955a27","Type":"ContainerStarted","Data":"a64f8f74a8537a8a41c23a1eb5e038452d8830aad17c0df59cd38fe345c874e0"} Oct 30 19:33:59.733127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.724797 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podStartSLOduration=1.724779091 podStartE2EDuration="1.724779091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:59.724519372 +0000 UTC m=+800.638123991" watchObservedRunningTime="2026-10-30 19:33:59.724779091 +0000 UTC m=+800.638383709" Oct 30 19:33:59.844185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.844150 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z"] Oct 30 19:33:59.859633 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.859602 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z"] Oct 30 19:33:59.859782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.859728 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.877003 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.876978 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-544fc8-openshift-default-dockercfg-t64zf\"" Oct 30 19:33:59.896329 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.896285 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.896329 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.896322 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.897199 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.896351 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.897199 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.896393 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08afa540-378d-4cc8-8a7c-7173ddffb742-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.897199 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.896415 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7wr\" (UniqueName: \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-kube-api-access-jl7wr\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.897199 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.896473 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.897199 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.896501 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.897199 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.896539 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.897199 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.896555 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.997417 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997339 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.997417 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997376 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.997655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997425 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.997655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997447 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.997655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997472 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.997655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997514 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08afa540-378d-4cc8-8a7c-7173ddffb742-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.997655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997530 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7wr\" (UniqueName: \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-kube-api-access-jl7wr\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.997655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997599 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.997949 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997665 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.997949 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997910 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.998064 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.997986 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.998160 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.998078 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.998217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.998169 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:33:59.998789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:33:59.998763 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08afa540-378d-4cc8-8a7c-7173ddffb742-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:34:00.000281 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.000258 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:34:00.000398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.000383 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:34:00.038122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.037801 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7wr\" (UniqueName: \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-kube-api-access-jl7wr\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:34:00.047159 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.038380 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:34:00.080424 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.080388 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:34:00.182877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.182849 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:34:00.340146 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.338515 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8k4b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:34:00.340146 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.338575 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" podUID="8da395a9-9be0-46ba-b397-ee54354bf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:34:00.557378 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.557292 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z"] Oct 30 19:34:00.569622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.568277 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:00.569622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.568357 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:00.569622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.568396 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:00.686230 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.686195 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" event={"ID":"08afa540-378d-4cc8-8a7c-7173ddffb742","Type":"ContainerStarted","Data":"ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651"} Oct 30 19:34:00.686384 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.686258 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" event={"ID":"08afa540-378d-4cc8-8a7c-7173ddffb742","Type":"ContainerStarted","Data":"5be8d397723f7908d87143c27702c788ee8c7cd63f0128326ed457f96f3020ad"} Oct 30 19:34:00.740542 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:00.740485 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" podStartSLOduration=1.740463946 podStartE2EDuration="1.740463946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:00.738275887 +0000 UTC m=+801.651880504" watchObservedRunningTime="2026-10-30 19:34:00.740463946 +0000 UTC m=+801.654068564" Oct 30 19:34:01.183022 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:01.182986 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:34:01.184224 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:01.184198 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:34:01.184340 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:01.184256 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:34:01.323500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:01.323456 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8k4b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:34:01.323689 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:01.323522 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" podUID="8da395a9-9be0-46ba-b397-ee54354bf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:34:02.183925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:02.183885 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:34:02.184356 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:02.183945 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:34:02.329137 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:02.328868 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8k4b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:34:02.329137 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:02.328926 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" podUID="8da395a9-9be0-46ba-b397-ee54354bf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:34:03.183699 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:03.183662 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:34:03.183875 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:03.183734 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:34:03.793165 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:03.793134 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:34:03.793655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:03.793433 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:34:03.794761 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:03.794738 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" Oct 30 19:34:04.184057 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:04.184025 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:34:04.184255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:04.184080 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:34:05.183420 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:05.183383 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:34:05.183839 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:05.183454 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:34:06.183596 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:06.183551 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:34:06.183986 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:06.183640 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:34:07.184131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:07.183502 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:34:07.184131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:07.183576 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:34:09.079171 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:09.079141 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:34:09.079665 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:09.079640 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:34:09.079945 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:09.079921 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:34:09.080724 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:09.080704 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:34:09.094255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:09.094232 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:34:09.811976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:09.811944 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:34:14.735718 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.735680 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc"] Oct 30 19:34:14.740907 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.740886 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.761144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.761116 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d797d2-openshift-default-dockercfg-qxcvg\"" Oct 30 19:34:14.777449 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.777419 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc"] Oct 30 19:34:14.838821 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.838793 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.838982 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.838827 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.838982 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.838861 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.839089 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.838997 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.839089 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.839062 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.839213 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.839089 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.839273 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.839237 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf8d8\" (UniqueName: \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-kube-api-access-sf8d8\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.839331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.839298 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.839388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.839374 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b09245-d0ba-4db8-b725-075c70cb3b90-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.940909 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.940874 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf8d8\" (UniqueName: \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-kube-api-access-sf8d8\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941072 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.940923 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941072 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.940975 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b09245-d0ba-4db8-b725-075c70cb3b90-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941072 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941036 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941072 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941064 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941340 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941144 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941340 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941199 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941340 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941241 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941340 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941273 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941409 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941578 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941542 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941652 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941805 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941787 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.941901 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.941858 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b09245-d0ba-4db8-b725-075c70cb3b90-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.943657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.943631 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.943896 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.943876 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.961508 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.961485 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:14.961711 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:14.961694 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf8d8\" (UniqueName: \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-kube-api-access-sf8d8\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:15.055926 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.055850 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:15.377091 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.377064 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc"] Oct 30 19:34:15.377873 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:34:15.377851 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b09245_d0ba_4db8_b725_075c70cb3b90.slice/crio-87579b0316e9a0e8af2b17b1d4329122982a80d16c615a877ea629ebe9fc6b9e WatchSource:0}: Error finding container 87579b0316e9a0e8af2b17b1d4329122982a80d16c615a877ea629ebe9fc6b9e: Status 404 returned error can't find the container with id 87579b0316e9a0e8af2b17b1d4329122982a80d16c615a877ea629ebe9fc6b9e Oct 30 19:34:15.380315 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.380281 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:15.380388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.380350 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:15.380388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.380376 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:15.732861 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.732830 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x"] Oct 30 19:34:15.737456 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.737430 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.757445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.757424 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4b4c34-openshift-default-dockercfg-w76ld\"" Oct 30 19:34:15.794540 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.788733 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x"] Oct 30 19:34:15.843721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.843679 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" event={"ID":"01b09245-d0ba-4db8-b725-075c70cb3b90","Type":"ContainerStarted","Data":"bdc7c17e06dc273b7a883794a23921dda54651d179f39da797a1ff1f779b650f"} Oct 30 19:34:15.843892 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.843736 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" event={"ID":"01b09245-d0ba-4db8-b725-075c70cb3b90","Type":"ContainerStarted","Data":"87579b0316e9a0e8af2b17b1d4329122982a80d16c615a877ea629ebe9fc6b9e"} Oct 30 19:34:15.853604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.853573 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c103d769-22e3-4f04-92f6-fc9677f3098b-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.853726 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.853618 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk4x6\" (UniqueName: \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-kube-api-access-dk4x6\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.853788 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.853731 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.853840 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.853798 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.853840 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.853826 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.853949 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.853843 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.854004 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.853960 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.854056 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.854021 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.854126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.854056 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.892186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.892121 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" podStartSLOduration=1.892085445 podStartE2EDuration="1.892085445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:15.889592895 +0000 UTC m=+816.803197526" watchObservedRunningTime="2026-10-30 19:34:15.892085445 +0000 UTC m=+816.805690063" Oct 30 19:34:15.958125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.957987 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.958125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.958047 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.958125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.958091 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.958415 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.958134 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.958415 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.958213 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.958415 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.958294 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.958415 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.958326 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.958627 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.958433 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c103d769-22e3-4f04-92f6-fc9677f3098b-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.958627 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.958463 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk4x6\" (UniqueName: \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-kube-api-access-dk4x6\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.959151 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.959126 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.959565 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.959543 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.960648 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.960588 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c103d769-22e3-4f04-92f6-fc9677f3098b-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.961033 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.961013 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.961373 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.961355 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.962529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.962502 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.963906 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.963888 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.982759 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.982732 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:15.983352 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:15.983310 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk4x6\" (UniqueName: \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-kube-api-access-dk4x6\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:16.055447 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.055408 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:16.056461 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.056434 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:16.058215 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.058176 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:34:16.058341 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.058238 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:34:16.392319 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.392294 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x"] Oct 30 19:34:16.394327 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:34:16.394298 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc103d769_22e3_4f04_92f6_fc9677f3098b.slice/crio-28deea3529374d7681f0bf5f6b5ee3fb354e431bb6e5f3f978a2e830fb2f998f WatchSource:0}: Error finding container 28deea3529374d7681f0bf5f6b5ee3fb354e431bb6e5f3f978a2e830fb2f998f: Status 404 returned error can't find the container with id 28deea3529374d7681f0bf5f6b5ee3fb354e431bb6e5f3f978a2e830fb2f998f Oct 30 19:34:16.397339 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.397307 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:16.397426 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.397375 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:16.397426 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.397408 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:16.859226 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.859185 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" event={"ID":"c103d769-22e3-4f04-92f6-fc9677f3098b","Type":"ContainerStarted","Data":"28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da"} Oct 30 19:34:16.861128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.859796 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" event={"ID":"c103d769-22e3-4f04-92f6-fc9677f3098b","Type":"ContainerStarted","Data":"28deea3529374d7681f0bf5f6b5ee3fb354e431bb6e5f3f978a2e830fb2f998f"} Oct 30 19:34:16.906066 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:16.906017 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" podStartSLOduration=1.9060034529999998 podStartE2EDuration="1.906003453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:16.904267912 +0000 UTC m=+817.817872546" watchObservedRunningTime="2026-10-30 19:34:16.906003453 +0000 UTC m=+817.819608070" Oct 30 19:34:17.059052 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:17.059017 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:34:17.059277 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:17.059189 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:34:17.059392 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:17.059034 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:17.061841 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:17.061780 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:34:17.061841 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:17.061829 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:34:18.059537 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:18.059500 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:34:18.060494 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:18.060465 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:34:18.060997 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:18.060395 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:34:18.061155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:18.061136 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:34:19.057128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:19.056478 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:34:19.057128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:19.056539 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:34:19.057396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:19.057198 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:34:19.057396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:19.057235 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:34:20.060128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:20.056464 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:34:20.060128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:20.056525 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:34:20.060128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:20.057401 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:34:20.060128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:20.057440 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:34:21.060128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:21.059634 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:34:21.060128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:21.059706 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:34:21.071128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:21.064142 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:34:21.071128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:21.064189 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:34:22.064527 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:22.064486 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:34:22.065021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:22.064550 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:34:22.074292 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:22.074259 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:34:22.074429 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:22.074312 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:34:23.056736 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.056688 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:34:23.057005 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.056764 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:34:23.063088 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.063056 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:23.063922 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.063904 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:23.064576 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.064554 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:34:23.704902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.704851 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj"] Oct 30 19:34:23.710979 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.710955 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.728708 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.728680 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d797d2\"/\"distinct-w1-d797d2-kserve-self-signed-certs\"" Oct 30 19:34:23.741042 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.740366 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj"] Oct 30 19:34:23.809127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.808395 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-model-cache\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.809127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.808458 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fgkm\" (UniqueName: \"kubernetes.io/projected/86e70905-a057-43ca-864c-52a11083e726-kube-api-access-2fgkm\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.809127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.808497 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e70905-a057-43ca-864c-52a11083e726-tls-certs\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.809127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.808556 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-tmp-dir\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.809127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.808595 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-home\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.809127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.808618 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-dshm\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.913999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.910414 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-model-cache\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.913999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.910464 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fgkm\" (UniqueName: \"kubernetes.io/projected/86e70905-a057-43ca-864c-52a11083e726-kube-api-access-2fgkm\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.913999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.910502 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e70905-a057-43ca-864c-52a11083e726-tls-certs\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.913999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.910559 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-tmp-dir\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.913999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.910594 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-home\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.913999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.910613 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-dshm\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.913999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.912902 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-model-cache\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.913999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.913943 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-home\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.914640 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.914273 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-tmp-dir\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.934257 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.923702 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e70905-a057-43ca-864c-52a11083e726-tls-certs\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.938127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.937477 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-dshm\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:23.942131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:23.941738 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fgkm\" (UniqueName: \"kubernetes.io/projected/86e70905-a057-43ca-864c-52a11083e726-kube-api-access-2fgkm\") pod \"distinct-w1-d797d2-kserve-7bc57b885f-h96rj\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:24.050762 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:24.050650 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:24.063158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:24.062821 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:24.063760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:24.063655 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:24.065042 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:24.065023 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:34:24.378032 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:24.378005 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj"] Oct 30 19:34:24.378371 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:34:24.378352 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e70905_a057_43ca_864c_52a11083e726.slice/crio-fae1b0d9b6f8f4692a02290606b02798b6ff38c0da7613682835fe6921597cc3 WatchSource:0}: Error finding container fae1b0d9b6f8f4692a02290606b02798b6ff38c0da7613682835fe6921597cc3: Status 404 returned error can't find the container with id fae1b0d9b6f8f4692a02290606b02798b6ff38c0da7613682835fe6921597cc3 Oct 30 19:34:24.512785 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:34:24.512755 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:34:25.001814 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:25.001778 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" event={"ID":"86e70905-a057-43ca-864c-52a11083e726","Type":"ContainerStarted","Data":"daee3efb808b85d571fe347857d2244eac9b9f5336982bcc6c160424fcd0ec56"} Oct 30 19:34:25.001995 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:25.001829 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" event={"ID":"86e70905-a057-43ca-864c-52a11083e726","Type":"ContainerStarted","Data":"fae1b0d9b6f8f4692a02290606b02798b6ff38c0da7613682835fe6921597cc3"} Oct 30 19:34:25.046800 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:25.046762 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podStartSLOduration=2.04674964 podStartE2EDuration="2.04674964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:25.046205942 +0000 UTC m=+825.959810552" watchObservedRunningTime="2026-10-30 19:34:25.04674964 +0000 UTC m=+825.960354257" Oct 30 19:34:25.051830 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:25.051806 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:30.273899 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.273858 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275"] Oct 30 19:34:30.280918 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.280891 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.296258 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.296234 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275"] Oct 30 19:34:30.305906 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.305883 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f913c6-openshift-default-dockercfg-m5wb4\"" Oct 30 19:34:30.358908 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.358870 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.358908 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.358903 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tg86\" (UniqueName: \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-kube-api-access-7tg86\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.359144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.359010 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.359144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.359047 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.359144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.359133 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50014489-db75-4a42-a19d-f9a14b3f7571-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.359308 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.359158 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.359308 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.359175 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50014489-db75-4a42-a19d-f9a14b3f7571-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.359308 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.359220 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.359308 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.359307 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.460871 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.460836 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.460871 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.460877 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tg86\" (UniqueName: \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-kube-api-access-7tg86\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461175 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.460954 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461175 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.460994 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461175 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.461030 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50014489-db75-4a42-a19d-f9a14b3f7571-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461175 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.461053 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461175 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.461077 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50014489-db75-4a42-a19d-f9a14b3f7571-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461175 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.461137 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461522 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.461195 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461522 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.461389 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461522 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.461490 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461679 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.461560 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461734 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.461682 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.461976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.461957 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50014489-db75-4a42-a19d-f9a14b3f7571-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.464594 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.464561 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.464710 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.464672 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50014489-db75-4a42-a19d-f9a14b3f7571-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.481603 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.481581 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.481808 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.481791 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tg86\" (UniqueName: \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-kube-api-access-7tg86\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.597024 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.596963 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:30.927561 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.927519 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275"] Oct 30 19:34:30.927970 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:34:30.927941 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50014489_db75_4a42_a19d_f9a14b3f7571.slice/crio-443fb1842daa5a7986ba5d373a40ae7885e7e8ea48742cc03e56b89b4fcecf0c WatchSource:0}: Error finding container 443fb1842daa5a7986ba5d373a40ae7885e7e8ea48742cc03e56b89b4fcecf0c: Status 404 returned error can't find the container with id 443fb1842daa5a7986ba5d373a40ae7885e7e8ea48742cc03e56b89b4fcecf0c Oct 30 19:34:30.931131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.931057 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:30.931229 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.931161 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:30.931229 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:30.931206 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:34:31.059316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:31.059271 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" event={"ID":"50014489-db75-4a42-a19d-f9a14b3f7571","Type":"ContainerStarted","Data":"d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196"} Oct 30 19:34:31.059489 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:31.059329 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" event={"ID":"50014489-db75-4a42-a19d-f9a14b3f7571","Type":"ContainerStarted","Data":"443fb1842daa5a7986ba5d373a40ae7885e7e8ea48742cc03e56b89b4fcecf0c"} Oct 30 19:34:31.112765 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:31.112719 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" podStartSLOduration=1.112706638 podStartE2EDuration="1.112706638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:31.109461281 +0000 UTC m=+832.023065900" watchObservedRunningTime="2026-10-30 19:34:31.112706638 +0000 UTC m=+832.026311318" Oct 30 19:34:31.597612 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:31.597577 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:31.599551 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:31.599521 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:34:31.599683 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:31.599584 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:34:32.598190 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:32.598153 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:34:32.598679 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:32.598212 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:34:33.597627 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:33.597591 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:34:33.597801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:33.597648 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:34:34.050994 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:34.050958 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:34.064012 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:34.063986 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:34.098751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:34.098727 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:34:34.597855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:34.597819 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:34:34.598015 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:34.597870 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:34:35.597569 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:35.597530 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:34:35.597953 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:35.597593 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:34:36.597496 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:36.597460 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:34:36.597663 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:36.597525 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:34:37.598041 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:37.597999 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:34:37.598515 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:37.598074 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:34:38.601435 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:38.601405 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:38.601901 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:38.601882 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:38.602617 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:38.602598 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:34:52.251775 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.251688 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt"] Oct 30 19:34:52.257751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.257728 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.277375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.277345 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt"] Oct 30 19:34:52.285494 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.285470 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-544fc8\"/\"distinct-w5-544fc8-kserve-self-signed-certs\"" Oct 30 19:34:52.374492 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.374455 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-home\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.374626 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.374513 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-dshm\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.374756 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.374722 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vn7k\" (UniqueName: \"kubernetes.io/projected/40606890-819d-40e3-ba76-cdd9c1be64fd-kube-api-access-2vn7k\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.374897 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.374788 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40606890-819d-40e3-ba76-cdd9c1be64fd-tls-certs\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.374897 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.374840 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-model-cache\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.374897 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.374879 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-tmp-dir\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.479928 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.479889 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-home\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.479928 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.479945 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-dshm\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.480265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.480041 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vn7k\" (UniqueName: \"kubernetes.io/projected/40606890-819d-40e3-ba76-cdd9c1be64fd-kube-api-access-2vn7k\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.480265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.480082 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40606890-819d-40e3-ba76-cdd9c1be64fd-tls-certs\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.480265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.480161 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-model-cache\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.480265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.480199 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-tmp-dir\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.480502 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.480421 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-home\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.480687 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.480665 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-tmp-dir\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.481342 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.481296 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-model-cache\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.483582 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.483556 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-dshm\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.483974 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.483928 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40606890-819d-40e3-ba76-cdd9c1be64fd-tls-certs\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.502719 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.502652 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vn7k\" (UniqueName: \"kubernetes.io/projected/40606890-819d-40e3-ba76-cdd9c1be64fd-kube-api-access-2vn7k\") pod \"distinct-w5-544fc8-kserve-5cfddc9444-hd7gt\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.596502 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.596465 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:52.919143 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:52.919115 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt"] Oct 30 19:34:52.919555 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:34:52.919531 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40606890_819d_40e3_ba76_cdd9c1be64fd.slice/crio-3375ff0ec6090c9a99e9eff6d58a4aaa78c391c586cbe1cc90c79e8d25bfc125 WatchSource:0}: Error finding container 3375ff0ec6090c9a99e9eff6d58a4aaa78c391c586cbe1cc90c79e8d25bfc125: Status 404 returned error can't find the container with id 3375ff0ec6090c9a99e9eff6d58a4aaa78c391c586cbe1cc90c79e8d25bfc125 Oct 30 19:34:53.253628 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:53.253588 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" event={"ID":"40606890-819d-40e3-ba76-cdd9c1be64fd","Type":"ContainerStarted","Data":"e6754d7e0f1590d578058551f8408effb6c5e10f920ebf47522bcfdd6387c343"} Oct 30 19:34:53.254153 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:53.253642 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" event={"ID":"40606890-819d-40e3-ba76-cdd9c1be64fd","Type":"ContainerStarted","Data":"3375ff0ec6090c9a99e9eff6d58a4aaa78c391c586cbe1cc90c79e8d25bfc125"} Oct 30 19:34:53.296178 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:53.296086 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podStartSLOduration=1.2960697890000001 podStartE2EDuration="1.296069789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:53.295410691 +0000 UTC m=+854.209015321" watchObservedRunningTime="2026-10-30 19:34:53.296069789 +0000 UTC m=+854.209674407" Oct 30 19:34:53.597088 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:34:53.597004 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:34:54.512911 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:34:54.512871 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:35:00.361075 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.361034 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx"] Oct 30 19:35:00.365656 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.365633 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.393160 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.393135 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx"] Oct 30 19:35:00.402343 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.402309 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd23d1\"/\"distinct-2-w3-dd23d1-kserve-self-signed-certs\"" Oct 30 19:35:00.480593 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.480562 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-tmp-dir\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.480773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.480694 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-home\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.480843 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.480774 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-model-cache\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.480843 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.480830 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-dshm\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.480934 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.480909 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk6x2\" (UniqueName: \"kubernetes.io/projected/5c198194-5eff-4b2f-9847-ac737f4caaf6-kube-api-access-kk6x2\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.481055 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.481038 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c198194-5eff-4b2f-9847-ac737f4caaf6-tls-certs\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.585156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.584510 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-tmp-dir\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.585156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.584630 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-home\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.585156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.584684 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-model-cache\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.585156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.584735 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-dshm\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.585156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.584768 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk6x2\" (UniqueName: \"kubernetes.io/projected/5c198194-5eff-4b2f-9847-ac737f4caaf6-kube-api-access-kk6x2\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.585156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.584931 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c198194-5eff-4b2f-9847-ac737f4caaf6-tls-certs\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.588798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.585803 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-tmp-dir\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.588798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.586094 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-home\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.622998 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.611860 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-model-cache\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.622998 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.614161 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-dshm\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.631773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.631739 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c198194-5eff-4b2f-9847-ac737f4caaf6-tls-certs\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.634180 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.633662 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk6x2\" (UniqueName: \"kubernetes.io/projected/5c198194-5eff-4b2f-9847-ac737f4caaf6-kube-api-access-kk6x2\") pod \"distinct-2-w3-dd23d1-kserve-676568fb49-6skzx\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.678412 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.678374 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:00.868255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.868224 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq"] Oct 30 19:35:00.874160 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.874083 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:00.890747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.890711 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq"] Oct 30 19:35:00.991598 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.991565 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e31c74-47bc-453f-81ef-795c60afeb3c-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:00.991598 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.991601 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:00.991836 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.991685 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:00.991836 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.991729 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:00.991972 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.991837 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8m2r\" (UniqueName: \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-kube-api-access-c8m2r\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:00.991972 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.991864 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:00.991972 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.991886 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:00.992140 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.992032 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:00.992140 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:00.992057 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.026356 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.026318 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx"] Oct 30 19:35:01.026647 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:35:01.026626 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c198194_5eff_4b2f_9847_ac737f4caaf6.slice/crio-12f6365db51aba4a7fbb92977e77f8a56f06c9b64bdba2270e17864a043d0d23 WatchSource:0}: Error finding container 12f6365db51aba4a7fbb92977e77f8a56f06c9b64bdba2270e17864a043d0d23: Status 404 returned error can't find the container with id 12f6365db51aba4a7fbb92977e77f8a56f06c9b64bdba2270e17864a043d0d23 Oct 30 19:35:01.093747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.093718 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.093863 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.093833 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.093923 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.093866 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.093984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.093968 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e31c74-47bc-453f-81ef-795c60afeb3c-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.094039 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.094011 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.094092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.094041 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.094092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.094082 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.094212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.094146 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.094212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.094170 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8m2r\" (UniqueName: \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-kube-api-access-c8m2r\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.094212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.094200 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.094570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.094547 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.094738 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.094594 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.094851 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.094798 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.094996 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.094973 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e31c74-47bc-453f-81ef-795c60afeb3c-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.097221 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.097198 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.097370 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.097348 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.115281 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.115258 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.115534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.115514 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8m2r\" (UniqueName: \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-kube-api-access-c8m2r\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.200146 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.200097 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:01.319092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.319060 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" event={"ID":"5c198194-5eff-4b2f-9847-ac737f4caaf6","Type":"ContainerStarted","Data":"a4ba9b33930532748ae7ae6a607ee5e2daf0c034fcd0cd56fb7b0b9ba513a98a"} Oct 30 19:35:01.319243 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.319130 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" event={"ID":"5c198194-5eff-4b2f-9847-ac737f4caaf6","Type":"ContainerStarted","Data":"12f6365db51aba4a7fbb92977e77f8a56f06c9b64bdba2270e17864a043d0d23"} Oct 30 19:35:01.366365 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.366320 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podStartSLOduration=1.366307071 podStartE2EDuration="1.366307071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:01.365471135 +0000 UTC m=+862.279075752" watchObservedRunningTime="2026-10-30 19:35:01.366307071 +0000 UTC m=+862.279911687" Oct 30 19:35:01.530516 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.530489 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq"] Oct 30 19:35:01.530949 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:35:01.530928 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e31c74_47bc_453f_81ef_795c60afeb3c.slice/crio-31ccaee3e6b1de528ae6976bae743bc86c1ebeade3b9cbdb15e007db552bc8dd WatchSource:0}: Error finding container 31ccaee3e6b1de528ae6976bae743bc86c1ebeade3b9cbdb15e007db552bc8dd: Status 404 returned error can't find the container with id 31ccaee3e6b1de528ae6976bae743bc86c1ebeade3b9cbdb15e007db552bc8dd Oct 30 19:35:01.533781 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.533743 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:01.533868 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.533814 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:01.533868 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.533847 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:01.678609 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:01.678578 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:02.355297 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:02.355261 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" event={"ID":"a5e31c74-47bc-453f-81ef-795c60afeb3c","Type":"ContainerStarted","Data":"e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c"} Oct 30 19:35:02.355518 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:02.355316 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" event={"ID":"a5e31c74-47bc-453f-81ef-795c60afeb3c","Type":"ContainerStarted","Data":"31ccaee3e6b1de528ae6976bae743bc86c1ebeade3b9cbdb15e007db552bc8dd"} Oct 30 19:35:02.448295 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:02.445773 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" podStartSLOduration=2.445748176 podStartE2EDuration="2.445748176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:02.431344059 +0000 UTC m=+863.344948665" watchObservedRunningTime="2026-10-30 19:35:02.445748176 +0000 UTC m=+863.359352793" Oct 30 19:35:02.597372 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:02.597326 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:35:02.610497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:02.610439 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:35:03.201117 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:03.201070 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:03.202514 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:03.202488 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:35:03.202640 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:03.202543 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:35:03.376611 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:03.376581 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:35:04.201325 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:04.201285 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:35:04.201689 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:04.201350 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:35:05.200880 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:05.200836 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:35:05.201088 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:05.200930 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:35:06.201028 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:06.200995 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:35:06.201449 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:06.201044 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:35:07.201020 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:07.200989 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:35:07.201225 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:07.201039 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:35:08.200731 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:08.200652 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:35:08.200731 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:08.200719 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:35:09.206353 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:09.206320 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:09.206813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:09.206794 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:09.207640 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:09.207620 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:35:10.678966 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:10.678932 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:10.693491 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:10.693465 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:11.500545 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:11.500514 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:14.859569 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.859539 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m"] Oct 30 19:35:14.864596 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.864571 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:14.888961 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.888937 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m"] Oct 30 19:35:14.914625 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.914575 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:14.914779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.914643 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:14.914779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.914675 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:14.914779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.914716 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:14.914957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.914824 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cba5790-7577-462a-aec0-759ff1d6bb40-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:14.914957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.914852 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rzc\" (UniqueName: \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-kube-api-access-f9rzc\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:14.914957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.914916 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:14.915136 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.914995 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:14.915136 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.915047 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:14.915249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.915161 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g"] Oct 30 19:35:14.919782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.919762 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:14.941914 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.941879 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g"] Oct 30 19:35:14.947039 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.947019 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h"] Oct 30 19:35:14.951791 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.951770 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:14.979216 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:14.977819 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h"] Oct 30 19:35:15.018810 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.018778 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.019038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.019018 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3793771e-d940-4326-b09d-288113928d46-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.019196 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.019181 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.019327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.019313 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.019458 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.019443 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.019622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.019608 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.019759 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.019743 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3793771e-d940-4326-b09d-288113928d46-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.019930 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.019903 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.019996 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.019960 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e564002-a4b1-4f14-9986-85fd4290caed-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.020050 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.019999 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.020050 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.019997 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.020189 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.020055 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.020356 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.020335 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.020423 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.020379 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.020482 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.020437 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwwd\" (UniqueName: \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-kube-api-access-jnwwd\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.020534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.020519 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.020658 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.020640 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.020841 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.020816 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2427l\" (UniqueName: \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-kube-api-access-2427l\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.020975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.020895 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.020975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.020922 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.020975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.020968 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.021179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.021094 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.021232 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.021189 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.021232 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.021217 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.021232 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.021218 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.021393 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.021297 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cba5790-7577-462a-aec0-759ff1d6bb40-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.021393 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.021341 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rzc\" (UniqueName: \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-kube-api-access-f9rzc\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.021493 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.021420 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.021493 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.021471 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.021591 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.021497 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e564002-a4b1-4f14-9986-85fd4290caed-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.022232 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.021933 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cba5790-7577-462a-aec0-759ff1d6bb40-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.022442 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.022184 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.024596 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.024573 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.024696 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.024591 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.040847 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.040822 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.047684 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.042939 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rzc\" (UniqueName: \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-kube-api-access-f9rzc\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.048046 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.048022 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6"] Oct 30 19:35:15.053387 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.053367 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.072465 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.072445 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6"] Oct 30 19:35:15.123911 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.123835 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.123911 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.123879 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.124156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.123924 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.123951 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124040 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87prz\" (UniqueName: \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-kube-api-access-87prz\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.124156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124079 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.124156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124149 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124173 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e564002-a4b1-4f14-9986-85fd4290caed-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124235 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124263 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.124479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124300 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124331 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3793771e-d940-4326-b09d-288113928d46-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.124479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124360 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.124479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124386 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.124479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124425 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124491 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3793771e-d940-4326-b09d-288113928d46-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124519 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124569 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124573 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124622 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048a2238-713a-4aa8-a6b1-d49457c0e66a-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124653 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e564002-a4b1-4f14-9986-85fd4290caed-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124675 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124722 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwwd\" (UniqueName: \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-kube-api-access-jnwwd\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124732 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124750 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124774 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124810 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2427l\" (UniqueName: \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-kube-api-access-2427l\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124821 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124835 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.124884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124874 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.125688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124900 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.125688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124929 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.125688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.124961 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.125688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.125017 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.125688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.125067 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e564002-a4b1-4f14-9986-85fd4290caed-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.125688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.125201 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3793771e-d940-4326-b09d-288113928d46-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.125688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.125548 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.128147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.127953 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3793771e-d940-4326-b09d-288113928d46-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.128147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.128061 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.128384 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.128361 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e564002-a4b1-4f14-9986-85fd4290caed-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.129514 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.129490 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.145633 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.145605 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.145740 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.145722 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.146507 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.146490 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2427l\" (UniqueName: \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-kube-api-access-2427l\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.146692 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.146671 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwwd\" (UniqueName: \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-kube-api-access-jnwwd\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.179150 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.179127 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:15.226223 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.226189 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.226409 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.226233 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.226409 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.226270 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.226409 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.226292 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.226409 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.226340 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.226409 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.226395 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87prz\" (UniqueName: \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-kube-api-access-87prz\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.226618 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.226444 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.226698 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.226677 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048a2238-713a-4aa8-a6b1-d49457c0e66a-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.226735 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.226722 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.227639 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.227606 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.227986 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.227961 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.228089 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.228027 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048a2238-713a-4aa8-a6b1-d49457c0e66a-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.228089 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.228056 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.228320 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.228213 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.230777 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.230752 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.232440 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.232421 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.236390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.236365 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:15.251319 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.251290 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.252264 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.252237 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87prz\" (UniqueName: \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-kube-api-access-87prz\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.286586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.286527 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:15.374580 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.374321 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:15.520165 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.519582 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m"] Oct 30 19:35:15.525084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.525042 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.525199 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.525140 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.525199 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.525181 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.544482 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.544450 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" event={"ID":"7cba5790-7577-462a-aec0-759ff1d6bb40","Type":"ContainerStarted","Data":"01571ce9ec9350b244dfe2ee5601a2988b9a133ef9f4d1081c4d0159911970bb"} Oct 30 19:35:15.614117 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:35:15.614070 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3793771e_d940_4326_b09d_288113928d46.slice/crio-17af1a86d380b38fa82950cb9be4a799bd35aa3e0f21f0fead1ff52ffdee72d2 WatchSource:0}: Error finding container 17af1a86d380b38fa82950cb9be4a799bd35aa3e0f21f0fead1ff52ffdee72d2: Status 404 returned error can't find the container with id 17af1a86d380b38fa82950cb9be4a799bd35aa3e0f21f0fead1ff52ffdee72d2 Oct 30 19:35:15.615438 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.615411 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g"] Oct 30 19:35:15.617121 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.617072 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.617233 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.617165 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.617233 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.617213 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.631959 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.631139 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h"] Oct 30 19:35:15.639169 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.639086 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.639248 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.639228 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.639298 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.639281 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.714712 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.714685 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6"] Oct 30 19:35:15.714975 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:35:15.714953 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048a2238_713a_4aa8_a6b1_d49457c0e66a.slice/crio-49a271541b6f1e200285ca0a491cff71a7bdfeece003248fb3ef9bdb8e452cbb WatchSource:0}: Error finding container 49a271541b6f1e200285ca0a491cff71a7bdfeece003248fb3ef9bdb8e452cbb: Status 404 returned error can't find the container with id 49a271541b6f1e200285ca0a491cff71a7bdfeece003248fb3ef9bdb8e452cbb Oct 30 19:35:15.718819 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.718778 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.718903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.718874 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:15.718965 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:15.718922 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:16.572729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.571993 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" event={"ID":"4e564002-a4b1-4f14-9986-85fd4290caed","Type":"ContainerStarted","Data":"ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd"} Oct 30 19:35:16.580367 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.580334 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" event={"ID":"4e564002-a4b1-4f14-9986-85fd4290caed","Type":"ContainerStarted","Data":"d63584e02ac3fbab25124304414110adba06217ae973c50ca08f796830cc9a95"} Oct 30 19:35:16.583981 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.583179 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" event={"ID":"3793771e-d940-4326-b09d-288113928d46","Type":"ContainerStarted","Data":"3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25"} Oct 30 19:35:16.583981 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.583231 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" event={"ID":"3793771e-d940-4326-b09d-288113928d46","Type":"ContainerStarted","Data":"17af1a86d380b38fa82950cb9be4a799bd35aa3e0f21f0fead1ff52ffdee72d2"} Oct 30 19:35:16.610135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.595112 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" event={"ID":"7cba5790-7577-462a-aec0-759ff1d6bb40","Type":"ContainerStarted","Data":"405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a"} Oct 30 19:35:16.610135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.598214 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" event={"ID":"048a2238-713a-4aa8-a6b1-d49457c0e66a","Type":"ContainerStarted","Data":"2dc46a6bc7554e9c8c2ceefcc5c27cde062f44050727dfd30512c86d1bd9ea97"} Oct 30 19:35:16.610135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.598257 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" event={"ID":"048a2238-713a-4aa8-a6b1-d49457c0e66a","Type":"ContainerStarted","Data":"49a271541b6f1e200285ca0a491cff71a7bdfeece003248fb3ef9bdb8e452cbb"} Oct 30 19:35:16.675242 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.671205 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podStartSLOduration=2.666081617 podStartE2EDuration="2.666081617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:16.648587307 +0000 UTC m=+877.562191926" watchObservedRunningTime="2026-10-30 19:35:16.666081617 +0000 UTC m=+877.579686235" Oct 30 19:35:16.726887 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.725610 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podStartSLOduration=2.725586067 podStartE2EDuration="2.725586067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:16.708607272 +0000 UTC m=+877.622211892" watchObservedRunningTime="2026-10-30 19:35:16.725586067 +0000 UTC m=+877.639190685" Oct 30 19:35:16.822845 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.822718 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podStartSLOduration=2.822696973 podStartE2EDuration="2.822696973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:16.77346158 +0000 UTC m=+877.687066199" watchObservedRunningTime="2026-10-30 19:35:16.822696973 +0000 UTC m=+877.736301591" Oct 30 19:35:16.824485 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:16.824440 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podStartSLOduration=1.824427865 podStartE2EDuration="1.824427865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:16.821132224 +0000 UTC m=+877.734736845" watchObservedRunningTime="2026-10-30 19:35:16.824427865 +0000 UTC m=+877.738032488" Oct 30 19:35:17.189197 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.189162 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:17.189577 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.189555 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:35:17.189758 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.189736 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:35:17.245318 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.245285 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:35:17.245547 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.245525 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:35:17.262588 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.262559 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:17.301124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.300568 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:35:17.301124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.300650 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:35:17.301124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.301060 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:17.381216 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.379170 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:17.381216 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.380233 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:17.381216 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.380288 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:35:17.512211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.506185 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:17.585271 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.585225 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:17.634350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.634303 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:17.932248 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:17.932193 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:18.022244 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.022192 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:18.181538 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.181498 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:35:18.181831 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.181798 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:35:18.239599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.239510 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:35:18.239831 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.239803 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:35:18.287835 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.287422 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:35:18.287835 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.287484 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:35:18.380239 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.380195 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:18.381475 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.381178 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:18.381916 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.381887 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:35:18.497271 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.492202 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:35:18.519126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.505183 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:18.593128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.592198 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:18.593128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.592847 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:35:18.637258 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.637197 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:18.637825 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.637797 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 19:35:18.638129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.638086 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:35:18.712315 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.712270 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:35:18.777192 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.773021 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:18.809239 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.809197 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:18.941288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.939298 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:18.941288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:18.940257 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:35:19.144181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.131588 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.456946 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.495963 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.528194 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.532241 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.532304 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.534735 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.534798 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.535346 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.535418 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.535472 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:35:19.545187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.535442 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:35:19.654394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.594177 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:19.654394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.627945 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:19.654394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.628005 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:35:19.723572 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.664191 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:19.739126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.725544 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:19.750607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.749727 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:35:19.778876 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.774162 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:19.778876 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.774453 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:35:19.825758 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.825134 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:35:19.844316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:19.826009 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:35:20.002882 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.001780 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:20.129694 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.129589 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:20.132073 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.132029 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:20.133750 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.133573 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:35:20.219536 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.211578 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:35:20.219536 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.211637 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:35:20.310761 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.310434 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:35:20.310761 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.310486 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:35:20.310761 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.310576 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:35:20.310761 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.310603 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:35:20.442064 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.442031 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:20.442354 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.442324 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:35:20.467766 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.467724 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:20.477255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.476087 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:35:20.504146 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.503510 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:20.568615 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.554505 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:20.657708 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.657664 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:35:20.664414 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.664373 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:20.741324 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.741218 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:20.779551 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.779513 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:20.844236 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:20.844190 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:21.009706 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.001951 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:21.138505 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.136387 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:35:21.161804 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.161761 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:21.306181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.245758 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:35:21.306181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.291616 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:35:21.306181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.276065 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:35:21.306181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.291712 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:35:21.306181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.305264 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:35:21.306181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.305319 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:35:21.395831 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.385547 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:21.395831 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.385632 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:35:21.457282 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.457234 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:21.499127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.477472 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:35:21.507129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.506180 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:21.575184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.559178 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:21.675131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.671934 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:35:21.694128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.685696 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:21.695428 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.694686 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:35:21.701555 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.701521 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:21.767014 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.766973 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:35:21.788736 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.788697 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:35:21.847278 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:21.847177 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:22.010128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.009137 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:35:22.048243 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.039382 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:35:22.076119 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.074886 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:35:22.241556 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.241504 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:35:22.255933 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.255896 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:35:22.256140 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.255969 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:35:22.259451 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.259425 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:35:22.285128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.280045 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:35:22.285128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.280146 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:35:22.305145 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.303903 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:35:22.305145 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.303974 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:35:22.422139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.410049 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:22.451677 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.451633 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:35:22.487147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.471981 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 19:35:22.487147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.472494 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:35:22.521011 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.520915 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:22.734727 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.692314 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:22.734727 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.694844 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:22.734727 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.732294 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:35:22.784383 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.784272 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:35:22.821523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.820960 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:35:22.848564 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:22.848523 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:23.010132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.009977 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:23.209186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.201063 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:35:23.209186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.201144 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:35:23.294130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.282960 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:23.295473 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.295441 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:35:23.295690 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.295662 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:35:23.297939 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.297917 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:35:23.298131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.298086 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:35:23.475313 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.472263 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:23.496377 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.492462 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:23.496377 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.492521 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:35:23.523259 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.523226 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:35:23.601928 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.575853 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:23.608519 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.602690 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:23.909066 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.908967 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:23.971396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:23.971356 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:24.031163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.012284 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:24.328127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.313018 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:24.462127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.432204 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:24.462127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.432270 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:35:24.462127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.435988 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:35:24.462127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.436050 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:35:24.462127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.459552 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:35:24.462127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.459702 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:35:24.462127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.460402 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:35:24.462127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.460467 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:35:24.486125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.475172 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:24.603477 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.601211 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:24.603477 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.601621 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:35:24.623218 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.622766 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:24.675891 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.672191 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:24.758128 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:35:24.734966 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:35:24.900182 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.896704 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:35:24.904093 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.904071 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:35:24.944263 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.944226 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:35:24.983063 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.982413 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:24.987122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.984267 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:35:24.989986 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:24.987985 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:35:25.003639 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:25.003616 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:35:25.006194 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:25.005579 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:35:25.182488 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:25.182425 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:35:25.182680 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:25.182557 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:35:25.237512 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:25.237120 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:35:25.237512 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:25.237175 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:35:25.287579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:25.287454 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:35:25.287579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:25.287510 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:35:25.386311 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:25.377563 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:25.386311 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:25.377623 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:35:26.247127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.247071 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:26.251246 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.248511 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:26.251246 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.249278 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:35:26.284746 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.284714 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:26.286040 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.286015 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:26.288078 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.288025 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:35:26.473655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.473623 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:26.475349 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.475325 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:26.476361 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.476299 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:35:26.812095 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.812066 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:26.813044 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.813019 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:26.814569 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:26.814549 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:35:30.623960 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.489046 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk"] Oct 30 19:35:30.712205 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.708323 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk"] Oct 30 19:35:30.712205 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.708490 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.809727 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.809638 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.820964 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.820933 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.821238 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.821219 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.821410 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.821395 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktzm\" (UniqueName: \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-kube-api-access-kktzm\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.822267 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.822246 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.822495 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.822479 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.822680 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.822667 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.822873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.822859 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.823014 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.823001 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.924203 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.924323 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.924353 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.924428 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.924483 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.924515 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.924539 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kktzm\" (UniqueName: \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-kube-api-access-kktzm\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.924577 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.924649 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925708 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.925130 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.925770 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.925717 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.929122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.926666 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.929122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.927507 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.929122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.927977 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.972864 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.967820 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.972864 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.968325 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.984989 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.984928 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktzm\" (UniqueName: \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-kube-api-access-kktzm\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:30.988147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:30.988125 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:31.048848 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:31.044781 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:31.413667 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:31.413573 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk"] Oct 30 19:35:31.423388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:31.423330 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:31.423663 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:31.423636 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:31.425818 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:31.425790 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:32.083366 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:32.083325 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" event={"ID":"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82","Type":"ContainerStarted","Data":"d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773"} Oct 30 19:35:32.083837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:32.083396 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" event={"ID":"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82","Type":"ContainerStarted","Data":"68729d1692b10a6f46cf31ca36b833ec7ebd5e49ea3b84d7477194ea7638bb36"} Oct 30 19:35:32.168128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:32.166749 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" podStartSLOduration=2.16672922 podStartE2EDuration="2.16672922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:32.150815026 +0000 UTC m=+893.064419649" watchObservedRunningTime="2026-10-30 19:35:32.16672922 +0000 UTC m=+893.080333837" Oct 30 19:35:33.052129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:33.051319 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:33.052129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:33.051466 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:35:33.052129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:33.051512 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:35:34.048470 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:34.047444 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:35:34.048470 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:34.047504 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:35:35.045833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:35.045797 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:35:35.046036 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:35.045872 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:35:36.045849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:36.045702 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:35:36.045849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:36.045759 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:35:37.047131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:37.046439 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:35:37.047131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:37.046499 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:35:38.046039 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:38.046002 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:35:38.046263 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:38.046074 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:35:39.411686 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:39.411656 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:35:39.412205 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:39.411656 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:35:39.434932 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:39.434907 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:35:39.435300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:39.435277 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:35:39.484805 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:39.484775 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:39.485475 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:39.485449 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:39.486363 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:39.486345 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:35:45.252259 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.252227 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8"] Oct 30 19:35:45.257888 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.257868 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.292134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.291365 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3735b2-openshift-default-dockercfg-z8mrd\"" Oct 30 19:35:45.292134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.292000 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8"] Oct 30 19:35:45.305529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.305496 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.305720 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.305576 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.305720 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.305646 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.305824 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.305760 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.305881 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.305826 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.305881 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.305855 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvk4w\" (UniqueName: \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-kube-api-access-vvk4w\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.306142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.305935 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.306142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.305996 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.306142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.306023 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.406934 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.406903 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407160 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.406961 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407160 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407002 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407160 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407027 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407160 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407082 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407160 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407142 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvk4w\" (UniqueName: \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-kube-api-access-vvk4w\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407207 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407267 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407294 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407582 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407466 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407669 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407650 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407698 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.407868 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407849 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.408009 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.407980 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.409622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.409604 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.409918 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.409900 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.428777 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.428754 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.429327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.429310 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvk4w\" (UniqueName: \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-kube-api-access-vvk4w\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.583432 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.583373 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:45.936180 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:35:45.936150 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014713a3_4a5c_49ff_8417_be78ee3a1cc2.slice/crio-45834312993f33db9d25523fcc5f1ca80eccbe8c82a1d6c5425587233e8944fb WatchSource:0}: Error finding container 45834312993f33db9d25523fcc5f1ca80eccbe8c82a1d6c5425587233e8944fb: Status 404 returned error can't find the container with id 45834312993f33db9d25523fcc5f1ca80eccbe8c82a1d6c5425587233e8944fb Oct 30 19:35:45.936619 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.936597 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8"] Oct 30 19:35:45.938655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.938624 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:45.938751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.938685 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:45.938751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:45.938712 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:46.013646 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.013622 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49"] Oct 30 19:35:46.019511 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.019488 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.044638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.044501 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49"] Oct 30 19:35:46.116046 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.116017 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.116240 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.116208 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.116313 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.116279 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.116365 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.116329 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.116470 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.116448 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.116579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.116564 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.116638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.116600 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.116760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.116740 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.116827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.116781 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wcjw\" (UniqueName: \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-kube-api-access-9wcjw\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.218094 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218016 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.218094 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218058 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wcjw\" (UniqueName: \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-kube-api-access-9wcjw\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.218332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218157 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.218332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218221 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.218332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218263 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.218332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218309 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.218541 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218337 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.218541 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218415 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.218668 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218641 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.218800 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218776 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.219193 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.219005 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.219193 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.219026 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.219193 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218808 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.219193 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.218782 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.221309 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.221286 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.221393 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.221302 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.241921 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.241896 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.244706 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.244685 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wcjw\" (UniqueName: \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-kube-api-access-9wcjw\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.252463 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.252437 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" event={"ID":"014713a3-4a5c-49ff-8417-be78ee3a1cc2","Type":"ContainerStarted","Data":"1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada"} Oct 30 19:35:46.252840 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.252488 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" event={"ID":"014713a3-4a5c-49ff-8417-be78ee3a1cc2","Type":"ContainerStarted","Data":"45834312993f33db9d25523fcc5f1ca80eccbe8c82a1d6c5425587233e8944fb"} Oct 30 19:35:46.308924 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.308876 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" podStartSLOduration=1.308863474 podStartE2EDuration="1.308863474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:46.303700786 +0000 UTC m=+907.217305404" watchObservedRunningTime="2026-10-30 19:35:46.308863474 +0000 UTC m=+907.222468134" Oct 30 19:35:46.342636 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.342597 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:46.584555 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.584463 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:46.683156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.683128 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49"] Oct 30 19:35:46.683819 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:35:46.683788 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c3a4f0_8862_43a6_89ac_3e6c847a6ab2.slice/crio-05120fdac870836dd4d711f82d611df3ec3356f416581fc2525e5baf535e99f6 WatchSource:0}: Error finding container 05120fdac870836dd4d711f82d611df3ec3356f416581fc2525e5baf535e99f6: Status 404 returned error can't find the container with id 05120fdac870836dd4d711f82d611df3ec3356f416581fc2525e5baf535e99f6 Oct 30 19:35:46.686875 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.686841 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:46.687021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.686908 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:46.687021 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:46.686939 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:35:47.275118 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:47.273240 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" event={"ID":"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2","Type":"ContainerStarted","Data":"0e181c0b871d188c29da9e93f01d0543d5aa0bf9869dbde5dd4413b69ef30ee3"} Oct 30 19:35:47.275118 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:47.273298 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" event={"ID":"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2","Type":"ContainerStarted","Data":"05120fdac870836dd4d711f82d611df3ec3356f416581fc2525e5baf535e99f6"} Oct 30 19:35:47.348590 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:47.348449 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:47.350167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:47.350078 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:35:47.350367 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:47.350140 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" podStartSLOduration=2.3501209960000002 podStartE2EDuration="2.350120996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:47.341595687 +0000 UTC m=+908.255200338" watchObservedRunningTime="2026-10-30 19:35:47.350120996 +0000 UTC m=+908.263725617" Oct 30 19:35:47.350367 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:47.350217 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:35:47.585746 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:47.585649 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:35:47.585746 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:47.585717 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:35:48.350169 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:48.349481 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:35:48.350169 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:48.349542 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:35:48.585123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:48.584512 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:35:48.585123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:48.584568 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:35:49.344137 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:49.343461 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:35:49.344137 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:49.343519 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:35:49.583696 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:49.583661 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:35:49.584075 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:49.583710 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:35:50.344134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:50.343663 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:35:50.344134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:50.343724 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:35:50.588081 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:50.587461 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:35:50.588081 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:50.587521 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:35:51.343145 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:51.343092 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:35:51.343379 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:51.343353 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:35:51.586434 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:51.586398 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:35:51.587667 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:51.587627 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:35:52.354132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:52.345475 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:35:52.354132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:52.345554 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:35:52.585594 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:52.585557 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:35:52.585765 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:52.585618 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:35:53.344007 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:53.343191 4904 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:35:53.344007 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:53.343277 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:35:54.076032 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:54.075993 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:54.076582 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:54.076395 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:54.077686 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:54.077664 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:35:54.518232 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:35:54.518174 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:35:54.944607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:54.944575 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:54.945333 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:54.945309 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:35:54.946080 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:35:54.946064 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:36:01.301130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.297178 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:01.521956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.520182 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:01.535127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.522374 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:01.566708 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.566613 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:01.619352 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.619308 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:36:01.682208 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.682165 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Oct 30 19:36:01.702414 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.702369 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:01.775130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.766513 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:01.803422 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.803384 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:36:01.852934 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.852810 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:36:01.910638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:01.910589 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:02.016254 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.016201 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:02.315222 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.315178 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:02.346127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.316433 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:36:02.423130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.420239 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 30 19:36:02.526226 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.526182 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:02.531133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.526666 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 19:36:02.554702 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.553675 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:36:02.555423 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.555405 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:36:02.650951 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.605559 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:36:02.651764 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.619813 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:02.652677 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.652380 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:36:02.653359 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.653343 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:36:02.739127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.709995 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:02.828083 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.779250 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:02.828903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.807177 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:02.829390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.829092 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:36:02.829599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.829585 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:36:02.920183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.868430 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:02.920183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.868798 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:36:02.988903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:02.988873 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:36:03.028226 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.028182 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:03.032229 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.028204 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:03.049369 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.049339 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:36:03.058512 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.058486 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:36:03.316327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.316287 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:36:03.464128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.459324 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:36:03.582179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.544237 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:03.603128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.562557 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:03.674527 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.674486 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 30 19:36:03.677779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.677122 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:36:03.837288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.837184 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:03.855127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.847939 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:36:03.855652 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.855619 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:36:03.877127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.859444 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:36:03.905128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:03.878222 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:04.067127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:04.063603 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:04.336125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:04.335245 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:04.476132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:04.469286 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:04.649179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:04.614877 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 19:36:04.649179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:04.615224 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:36:04.702628 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:04.682602 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:36:04.860233 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:04.860186 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:36:04.860785 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:04.860756 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:36:04.886813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:04.886520 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:04.896148 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:04.895229 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:05.004126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.002079 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:05.035575 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.035544 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:36:05.063604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.063569 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:05.064079 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.064054 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:05.105072 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.104592 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:36:05.111851 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.111827 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:36:05.112787 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.112768 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:36:05.121533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.121462 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:36:05.145459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.145421 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:36:05.171861 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.171826 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:36:05.173842 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.173818 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:36:05.193765 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:05.193740 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:36:24.511724 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:36:24.511645 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:36:30.644129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.643160 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh"] Oct 30 19:36:30.654564 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.654538 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:30.665946 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.665925 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh"] Oct 30 19:36:30.690332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.690306 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8"] Oct 30 19:36:30.715472 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.715448 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:30.793286 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.793163 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8"] Oct 30 19:36:30.823976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.823940 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059991be-f076-488f-9aa1-70521ef6460a-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826272 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826358 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826396 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnzl5\" (UniqueName: \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-kube-api-access-fnzl5\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826429 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826471 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059991be-f076-488f-9aa1-70521ef6460a-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826529 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826580 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826627 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5nbc\" (UniqueName: \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-kube-api-access-t5nbc\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826657 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826711 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826753 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826791 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826837 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826918 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.826944 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.827018 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:30.835280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:30.827044 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.187230 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.183301 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:31.218125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.193140 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.254725 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzl5\" (UniqueName: \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-kube-api-access-fnzl5\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262317 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262383 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059991be-f076-488f-9aa1-70521ef6460a-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262450 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262513 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262567 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5nbc\" (UniqueName: \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-kube-api-access-t5nbc\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262607 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262666 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262723 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262772 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262828 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262920 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.262953 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.263037 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.263071 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.263176 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059991be-f076-488f-9aa1-70521ef6460a-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.266130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.263276 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.292151 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.283502 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.292151 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.285392 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.342171 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.327291 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.382161 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.348890 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059991be-f076-488f-9aa1-70521ef6460a-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.436646 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.436611 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnzl5\" (UniqueName: \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-kube-api-access-fnzl5\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.490177 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.480775 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.517312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.491725 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.517312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.496684 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.517312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.497293 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.518011 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.517941 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:31.679128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.676652 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:31.781175 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.736722 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.781175 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.750573 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:31.791133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.787228 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.832800 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.832709 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059991be-f076-488f-9aa1-70521ef6460a-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.936601 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.936566 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:31.987356 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:31.987304 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:36:32.018148 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.018092 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:32.181091 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.179697 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:32.181091 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.180809 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:32.192133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.188755 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:32.220124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.195151 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw"] Oct 30 19:36:32.250918 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.250885 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:36:32.252475 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.252425 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:32.254799 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.254760 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:32.275163 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.255243 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:36:32.277641 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.265662 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:32.341287 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.341253 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw"] Oct 30 19:36:32.341599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.341581 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:36:32.343376 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.343358 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.354026 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.354002 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:32.436964 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.436876 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.437298 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.437274 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.440302 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.440278 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.445134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.440536 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fm9p\" (UniqueName: \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-kube-api-access-5fm9p\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.563155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.549536 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.563155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.549703 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.563155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.549781 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.563155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.549818 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.563155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.549907 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.563155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.545536 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:36:32.563155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.551425 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:36:32.570158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.570127 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5nbc\" (UniqueName: \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-kube-api-access-t5nbc\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:32.643127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.632659 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:32.719564 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.709739 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.755181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.754463 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:32.765327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.754969 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:32.890210 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.856048 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:36:32.958274 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.958239 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.961051 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.961205 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.987571 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.987822 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.988592 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.989703 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fm9p\" (UniqueName: \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-kube-api-access-5fm9p\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.989767 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.989909 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.989980 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.990014 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.990079 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.991543 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.991803 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.992491 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.027185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:32.992736 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.081813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.081785 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.082026 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.081851 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:33.083222 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.083198 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.115137 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.109807 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fm9p\" (UniqueName: \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-kube-api-access-5fm9p\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.175586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.175557 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:36:33.187527 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.181669 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:33.211827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.188054 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:36:33.211827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.181703 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:36:33.211827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.189580 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:36:33.255940 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.255557 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:33.256530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.256499 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:36:33.269390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.268990 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:33.345893 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.345570 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:33.349267 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.348182 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.408919 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.408887 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:36:33.410260 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.410240 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:36:33.410448 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.410435 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:36:33.410566 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.410556 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:36:33.410669 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.410660 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:36:33.418142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.418122 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:36:33.477438 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.477402 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:36:33.492926 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.492667 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:33.754257 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.754232 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:36:33.965678 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.965655 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:36:34.005957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.985334 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:36:34.005957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.985491 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:36:34.005957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.985592 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:36:34.005957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.986528 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:36:34.005957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.986600 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:36:34.005957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:33.986810 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8"] Oct 30 19:36:34.283712 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:36:34.283681 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059991be_f076_488f_9aa1_70521ef6460a.slice/crio-5f8aa4805edd7f76a7ffbd9895fae48fe9eb7091197e28bc2882c6323a3c45b4 WatchSource:0}: Error finding container 5f8aa4805edd7f76a7ffbd9895fae48fe9eb7091197e28bc2882c6323a3c45b4: Status 404 returned error can't find the container with id 5f8aa4805edd7f76a7ffbd9895fae48fe9eb7091197e28bc2882c6323a3c45b4 Oct 30 19:36:34.284188 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.284166 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh"] Oct 30 19:36:34.286863 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.286790 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:36:34.286986 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.286925 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:36:34.287157 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.287011 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:36:34.407709 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.407685 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw"] Oct 30 19:36:34.409797 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:36:34.409771 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aaa2c8b_0e42_4d09_9b9c_d18fce9f7d65.slice/crio-af21396814e72ffcd45fbf24b7ea391c5ede08cfdc83a5574101676da61928c3 WatchSource:0}: Error finding container af21396814e72ffcd45fbf24b7ea391c5ede08cfdc83a5574101676da61928c3: Status 404 returned error can't find the container with id af21396814e72ffcd45fbf24b7ea391c5ede08cfdc83a5574101676da61928c3 Oct 30 19:36:34.413520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.413485 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:36:34.413690 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.413552 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:36:34.413690 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.413583 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:36:34.524330 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.524298 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" event={"ID":"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65","Type":"ContainerStarted","Data":"af21396814e72ffcd45fbf24b7ea391c5ede08cfdc83a5574101676da61928c3"} Oct 30 19:36:34.527999 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.527974 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" event={"ID":"059991be-f076-488f-9aa1-70521ef6460a","Type":"ContainerStarted","Data":"6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b"} Oct 30 19:36:34.530076 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.530044 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" event={"ID":"059991be-f076-488f-9aa1-70521ef6460a","Type":"ContainerStarted","Data":"5f8aa4805edd7f76a7ffbd9895fae48fe9eb7091197e28bc2882c6323a3c45b4"} Oct 30 19:36:34.541684 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.541661 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" event={"ID":"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7","Type":"ContainerStarted","Data":"d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09"} Oct 30 19:36:34.541901 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.541880 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" event={"ID":"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7","Type":"ContainerStarted","Data":"e78618b7b9918de2184715b408b3eacca7c43e774782a09e0cfaf41ea2ae81a8"} Oct 30 19:36:34.598944 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.598836 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" podStartSLOduration=4.598820072 podStartE2EDuration="4.598820072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:34.597429415 +0000 UTC m=+955.511034033" watchObservedRunningTime="2026-10-30 19:36:34.598820072 +0000 UTC m=+955.512424691" Oct 30 19:36:34.961192 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.961160 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:34.961790 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:34.961768 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:35.894212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:35.833702 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" event={"ID":"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65","Type":"ContainerStarted","Data":"14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86"} Oct 30 19:36:36.003137 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:35.960510 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podStartSLOduration=5.960492203 podStartE2EDuration="5.960492203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:34.649040213 +0000 UTC m=+955.562644830" watchObservedRunningTime="2026-10-30 19:36:35.960492203 +0000 UTC m=+956.874096821" Oct 30 19:36:36.019161 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.013681 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:36:36.019161 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.013736 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 19:36:36.206133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.197081 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:36:36.206459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.206424 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 19:36:36.233128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.216028 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" podStartSLOduration=6.216005348 podStartE2EDuration="6.216005348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:36.011867343 +0000 UTC m=+956.925471962" watchObservedRunningTime="2026-10-30 19:36:36.216005348 +0000 UTC m=+957.129609969" Oct 30 19:36:36.497555 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.497473 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:36.499762 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.499732 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:36:36.508557 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.499934 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:36:36.632251 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.632209 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:36.907478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.907384 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:36.964493 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.964457 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:36:36.964753 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.964724 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 19:36:36.965472 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.965449 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:36:36.965639 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:36.965618 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 19:36:37.018917 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:37.018880 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:37.328822 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:37.328214 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:37.505165 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:37.496946 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:36:37.505165 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:37.497009 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:36:37.513262 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:37.513218 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:37.877129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:37.849335 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:36:37.965502 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:37.965461 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:36:37.965777 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:37.965748 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 19:36:37.975537 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:37.968253 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:36:37.975537 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:37.968299 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 19:36:38.027130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.020235 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:38.027130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.020725 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:36:38.070165 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.063373 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:38.258484 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.246825 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:36:38.426452 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.426382 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:38.490993 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.490188 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:38.519173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.518046 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:36:38.519173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.518137 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:36:38.624174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.621799 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:38.624174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.622094 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 19:36:38.853191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.852250 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:38.853373 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:38.853344 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:36:39.007314 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.007258 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:36:39.007584 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.007554 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 19:36:39.008690 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.008667 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:36:39.008857 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.008837 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 19:36:39.011628 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.011602 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:39.032748 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.032709 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:39.092262 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.072663 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:39.376489 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.370056 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:39.486530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.478438 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:39.522159 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.513631 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:36:39.522159 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.513686 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:36:39.522159 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.515590 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:39.600600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.600566 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:36:39.600875 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.600859 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:36:39.601010 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.600999 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:36:39.625216 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.625172 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:39.626053 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.626029 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:36:39.854402 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.854360 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:36:39.872225 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.872167 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:39.882558 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.882532 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:36:39.896170 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:39.896144 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:36:40.011584 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.011538 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:36:40.094136 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.014078 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 19:36:40.094136 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.042429 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:40.096358 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.070918 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:36:40.096358 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.083829 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:36:40.096577 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.096379 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 19:36:40.108467 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.108379 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:40.110325 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.110304 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:36:40.145310 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.145262 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:40.489539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.489187 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:40.649128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.642157 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:40.649128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.642723 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:36:40.649128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.642764 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:36:40.858718 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:40.858631 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:41.077136 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.077075 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:36:41.077448 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.077414 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 19:36:41.113184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.087452 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:41.151130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.150342 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 19:36:41.151130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.150454 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:36:41.151130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.150836 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:36:41.176127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.152909 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:36:41.176127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.152954 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 19:36:41.176127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.153465 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:41.176127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.159256 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:41.359124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.358481 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:36:41.510145 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.504494 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:41.528020 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.527984 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:36:41.544722 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.544686 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:36:41.642434 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.642394 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:41.687124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.685446 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:36:41.687124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.685515 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:36:41.687124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.686751 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:41.695126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.691301 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:36:41.695126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.694165 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:36:41.736638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.736609 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:36:41.747489 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.747465 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:36:41.964498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.964462 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:36:41.964726 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.964519 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 19:36:41.964899 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.964870 4904 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:36:41.964995 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:41.964916 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 19:36:42.498442 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:42.498393 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:36:42.498853 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:42.498459 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:36:43.369454 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:43.369323 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:43.370937 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:43.370445 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:43.372328 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:43.371933 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:36:43.533581 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:43.533069 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:43.534779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:43.534569 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:43.536661 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:43.536594 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:36:43.677951 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:43.677918 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:43.679147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:43.679069 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:43.680594 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:43.680573 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:36:48.953778 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:48.953534 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp"] Oct 30 19:36:48.959880 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:48.959855 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:48.967945 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:48.967905 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp"] Oct 30 19:36:48.976347 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:48.976324 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:48.976529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:48.976506 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4qb5\"" Oct 30 19:36:49.025330 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.025294 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4mf\" (UniqueName: \"kubernetes.io/projected/83190974-909e-4e8e-869e-1db259d1fe17-kube-api-access-nv4mf\") pod \"maas-api-revoke-sub-7d3ed095-h6xdp\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:49.025494 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.025463 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d3ed095-h6xdp\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:49.025494 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.025491 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d3ed095-h6xdp\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:49.126291 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.126262 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d3ed095-h6xdp\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:49.126438 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.126302 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d3ed095-h6xdp\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:49.126499 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.126461 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4mf\" (UniqueName: \"kubernetes.io/projected/83190974-909e-4e8e-869e-1db259d1fe17-kube-api-access-nv4mf\") pod \"maas-api-revoke-sub-7d3ed095-h6xdp\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:49.127241 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.127214 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d3ed095-h6xdp\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:49.128793 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.128770 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d3ed095-h6xdp\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:49.150504 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.150466 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4mf\" (UniqueName: \"kubernetes.io/projected/83190974-909e-4e8e-869e-1db259d1fe17-kube-api-access-nv4mf\") pod \"maas-api-revoke-sub-7d3ed095-h6xdp\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:49.278962 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.278888 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:49.592051 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.592021 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp"] Oct 30 19:36:49.592714 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:36:49.592692 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83190974_909e_4e8e_869e_1db259d1fe17.slice/crio-39dcd95e41d8e7678f24f3e91b88c8c9a614aae13071a845092c78a671a24498 WatchSource:0}: Error finding container 39dcd95e41d8e7678f24f3e91b88c8c9a614aae13071a845092c78a671a24498: Status 404 returned error can't find the container with id 39dcd95e41d8e7678f24f3e91b88c8c9a614aae13071a845092c78a671a24498 Oct 30 19:36:49.595123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:49.595071 4904 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:36:50.399658 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:50.399550 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" event={"ID":"83190974-909e-4e8e-869e-1db259d1fe17","Type":"ContainerStarted","Data":"39dcd95e41d8e7678f24f3e91b88c8c9a614aae13071a845092c78a671a24498"} Oct 30 19:36:52.416828 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:52.416794 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" event={"ID":"83190974-909e-4e8e-869e-1db259d1fe17","Type":"ContainerStarted","Data":"330d1bafafd0b5c2b67525c0c5ad4683f0e8bc960927806c1958b95793638182"} Oct 30 19:36:52.462396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:52.462349 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" podStartSLOduration=2.011980829 podStartE2EDuration="4.462334285s" totalImagesPullingTime="2.450353456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:52.458381712 +0000 UTC m=+973.371986329" watchObservedRunningTime="2026-10-30 19:36:52.462334285 +0000 UTC m=+973.375938902" Oct 30 19:36:53.429149 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:53.429093 4904 generic.go:356] "Generic (PLEG): container finished" podID="83190974-909e-4e8e-869e-1db259d1fe17" containerID="330d1bafafd0b5c2b67525c0c5ad4683f0e8bc960927806c1958b95793638182" exitCode=0 Oct 30 19:36:53.429642 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:53.429247 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" event={"ID":"83190974-909e-4e8e-869e-1db259d1fe17","Type":"ContainerDied","Data":"330d1bafafd0b5c2b67525c0c5ad4683f0e8bc960927806c1958b95793638182"} Oct 30 19:36:54.548799 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:36:54.548756 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:36:54.616388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:54.616357 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:54.715610 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:54.715522 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83190974-909e-4e8e-869e-1db259d1fe17-kube-api-access-nv4mf\" (UniqueName: \"kubernetes.io/projected/83190974-909e-4e8e-869e-1db259d1fe17-kube-api-access-nv4mf\") pod \"83190974-909e-4e8e-869e-1db259d1fe17\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " Oct 30 19:36:54.715789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:54.715657 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-xks-ca\") pod \"83190974-909e-4e8e-869e-1db259d1fe17\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " Oct 30 19:36:54.715789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:54.715767 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-openshift-ca\") pod \"83190974-909e-4e8e-869e-1db259d1fe17\" (UID: \"83190974-909e-4e8e-869e-1db259d1fe17\") " Oct 30 19:36:54.716198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:54.716171 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-openshift-ca" pod "83190974-909e-4e8e-869e-1db259d1fe17" (UID: "83190974-909e-4e8e-869e-1db259d1fe17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:54.716340 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:54.716313 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-openshift-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:36:54.718551 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:54.718520 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83190974-909e-4e8e-869e-1db259d1fe17-kube-api-access-nv4mf" pod "83190974-909e-4e8e-869e-1db259d1fe17" (UID: "83190974-909e-4e8e-869e-1db259d1fe17"). InnerVolumeSpecName "kube-api-access-nv4mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:54.718709 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:54.718693 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-xks-ca" pod "83190974-909e-4e8e-869e-1db259d1fe17" (UID: "83190974-909e-4e8e-869e-1db259d1fe17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:54.817886 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:54.817843 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv4mf\" (UniqueName: \"kubernetes.io/projected/83190974-909e-4e8e-869e-1db259d1fe17-kube-api-access-nv4mf\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:36:54.817886 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:54.817883 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83190974-909e-4e8e-869e-1db259d1fe17-maas-api-xks-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:36:55.449316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:55.449274 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" event={"ID":"83190974-909e-4e8e-869e-1db259d1fe17","Type":"ContainerDied","Data":"39dcd95e41d8e7678f24f3e91b88c8c9a614aae13071a845092c78a671a24498"} Oct 30 19:36:55.449316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:55.449292 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp" Oct 30 19:36:55.449316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:55.449321 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39dcd95e41d8e7678f24f3e91b88c8c9a614aae13071a845092c78a671a24498" Oct 30 19:36:59.620154 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.620121 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7254-b0200d"] Oct 30 19:36:59.620733 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.620718 4904 cpu_manager.go:414] "containerMap: removing container" podUID="83190974-909e-4e8e-869e-1db259d1fe17" containerName="revoke-keys" Oct 30 19:36:59.620780 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.620735 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83190974-909e-4e8e-869e-1db259d1fe17" containerName="revoke-keys" Oct 30 19:36:59.620859 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.620849 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="83190974-909e-4e8e-869e-1db259d1fe17" containerName="revoke-keys" Oct 30 19:36:59.624747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.624729 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:36:59.641666 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.641639 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7254-b0200d"] Oct 30 19:36:59.648646 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.648624 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r79h9\"" Oct 30 19:36:59.795060 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.795023 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vq9j\" (UniqueName: \"kubernetes.io/projected/d5e3e7cd-6367-4413-9969-ee275ab60bfc-kube-api-access-2vq9j\") pod \"test-curl-7254-b0200d\" (UID: \"d5e3e7cd-6367-4413-9969-ee275ab60bfc\") " pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:36:59.896901 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.896814 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vq9j\" (UniqueName: \"kubernetes.io/projected/d5e3e7cd-6367-4413-9969-ee275ab60bfc-kube-api-access-2vq9j\") pod \"test-curl-7254-b0200d\" (UID: \"d5e3e7cd-6367-4413-9969-ee275ab60bfc\") " pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:36:59.918538 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.918515 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vq9j\" (UniqueName: \"kubernetes.io/projected/d5e3e7cd-6367-4413-9969-ee275ab60bfc-kube-api-access-2vq9j\") pod \"test-curl-7254-b0200d\" (UID: \"d5e3e7cd-6367-4413-9969-ee275ab60bfc\") " pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:36:59.939736 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:36:59.939709 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:37:00.253050 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.253016 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7254-b0200d"] Oct 30 19:37:00.253657 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:37:00.253634 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5e3e7cd_6367_4413_9969_ee275ab60bfc.slice/crio-5779bb8a36d20158d45afc16b0d871021cc420b93252e9004f2bebaf4d4283e6 WatchSource:0}: Error finding container 5779bb8a36d20158d45afc16b0d871021cc420b93252e9004f2bebaf4d4283e6: Status 404 returned error can't find the container with id 5779bb8a36d20158d45afc16b0d871021cc420b93252e9004f2bebaf4d4283e6 Oct 30 19:37:00.498733 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.498701 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7254-b0200d" event={"ID":"d5e3e7cd-6367-4413-9969-ee275ab60bfc","Type":"ContainerStarted","Data":"5779bb8a36d20158d45afc16b0d871021cc420b93252e9004f2bebaf4d4283e6"} Oct 30 19:37:00.502490 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.502426 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8"] Oct 30 19:37:00.508331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.508308 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.538581 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.537254 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8"] Oct 30 19:37:00.582241 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.582214 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt"] Oct 30 19:37:00.587723 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.587700 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.598991 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.598705 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q"] Oct 30 19:37:00.611139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.610926 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.612224 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.612195 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt"] Oct 30 19:37:00.618234 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.618211 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.618532 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.618508 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.619394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.618576 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.619394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.618691 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5d54\" (UniqueName: \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-kube-api-access-r5d54\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.619394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.618742 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.619394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.618792 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.619394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.618953 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.619394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.619007 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.619394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.619069 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.631970 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.631941 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q"] Oct 30 19:37:00.721139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721089 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.721139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721142 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.721375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721266 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.721375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721318 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.721375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721366 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.721533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721399 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.721533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721470 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.721533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721518 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.721676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721537 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.721676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721618 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.721773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721712 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.721773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721763 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.721862 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721804 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.721911 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721887 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.722015 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721918 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.722015 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721947 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.722015 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.721948 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjhrh\" (UniqueName: \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-kube-api-access-gjhrh\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.722015 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722005 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.722227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722035 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47126122-ef5b-4e3f-9449-2fee7766f55a-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.722227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722081 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.722227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722134 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.722227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722162 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.722420 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722226 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqz7\" (UniqueName: \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-kube-api-access-8lqz7\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.722420 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722266 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.722420 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722289 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.722420 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722343 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.722420 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722373 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5d54\" (UniqueName: \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-kube-api-access-r5d54\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.722729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722423 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.722729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.722470 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.723154 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.723093 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.723288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.723195 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.724005 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.723978 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.728479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.728453 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.728768 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.728746 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.742585 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.742564 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.742664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.742629 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5d54\" (UniqueName: \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-kube-api-access-r5d54\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.823411 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823383 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.823517 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823435 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.823517 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823503 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.823697 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823537 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:00.823697 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823582 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.823697 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823618 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjhrh\" (UniqueName: \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-kube-api-access-gjhrh\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.823697 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823656 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47126122-ef5b-4e3f-9449-2fee7766f55a-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.823903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823707 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.823903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823733 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.823903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823787 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqz7\" (UniqueName: \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-kube-api-access-8lqz7\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.823903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823824 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.823903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823850 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.823903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823896 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.824456 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823949 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.824456 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.823973 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.824456 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.824031 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.824456 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.824075 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.824456 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.824144 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.824456 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.824186 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.824456 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.824434 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47126122-ef5b-4e3f-9449-2fee7766f55a-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.824962 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.824559 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.826431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.825163 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.826431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.825485 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.826431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.825648 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.826431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.825739 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.826431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.825792 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.826431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.826376 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.826804 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.826441 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.826804 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.826680 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.832450 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.832266 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.832450 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.832416 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.832764 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.832738 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.833605 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.833585 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.845206 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.845187 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjhrh\" (UniqueName: \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-kube-api-access-gjhrh\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.845721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.845700 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.846079 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.846056 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqz7\" (UniqueName: \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-kube-api-access-8lqz7\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.846176 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.846071 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:00.929385 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.929359 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:00.943985 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:00.943954 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:01.251303 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.240564 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:01.251303 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.240646 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:01.251303 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.240690 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:01.251303 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.248380 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8"] Oct 30 19:37:01.511936 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.511890 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q"] Oct 30 19:37:01.640800 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.638817 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:01.640800 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.638914 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:01.640800 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.638958 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:01.695135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.670370 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" event={"ID":"65b61c1d-ecfb-484b-bc72-6b644dd10b8d","Type":"ContainerStarted","Data":"cd5d979fcfdda3a583040875a25390ec5affb4aaede5f55759c516b478640d3d"} Oct 30 19:37:01.718039 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.717937 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt"] Oct 30 19:37:01.734061 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.733215 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7254-b0200d" event={"ID":"d5e3e7cd-6367-4413-9969-ee275ab60bfc","Type":"ContainerStarted","Data":"c3c7338fd3ff2a96046bf18fabf9d149f5f8952a3b1fa323bb9db0f33d49901b"} Oct 30 19:37:01.819492 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.811606 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podStartSLOduration=1.811583953 podStartE2EDuration="1.811583953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:01.754432083 +0000 UTC m=+982.668036704" watchObservedRunningTime="2026-10-30 19:37:01.811583953 +0000 UTC m=+982.725188571" Oct 30 19:37:01.819492 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.812210 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7254-b0200d" podStartSLOduration=2.317853498 podStartE2EDuration="2.812200322s" totalImagesPullingTime="494.346824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:01.799445735 +0000 UTC m=+982.713050352" watchObservedRunningTime="2026-10-30 19:37:01.812200322 +0000 UTC m=+982.725804951" Oct 30 19:37:01.821075 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.820116 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:01.821075 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.820209 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:01.821075 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.820257 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:01.837350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:01.836864 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:02.843169 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:02.833512 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:02.843169 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:02.833585 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:02.941335 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:02.941295 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" event={"ID":"65b61c1d-ecfb-484b-bc72-6b644dd10b8d","Type":"ContainerStarted","Data":"e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f"} Oct 30 19:37:02.958949 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:02.958912 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" event={"ID":"47126122-ef5b-4e3f-9449-2fee7766f55a","Type":"ContainerStarted","Data":"53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5"} Oct 30 19:37:02.959221 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:02.959203 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" event={"ID":"47126122-ef5b-4e3f-9449-2fee7766f55a","Type":"ContainerStarted","Data":"5ee4b6041bfc425c881d85477d79e5601a9a5e542f18d7ab18c1000f4be0c07a"} Oct 30 19:37:02.975115 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:02.966509 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:02.975115 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:02.972619 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" event={"ID":"517ef3f2-04c5-4a53-8721-c5e44a9f494f","Type":"ContainerStarted","Data":"849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf"} Oct 30 19:37:02.975115 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:02.972684 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" event={"ID":"517ef3f2-04c5-4a53-8721-c5e44a9f494f","Type":"ContainerStarted","Data":"51b4c6083b7b98b565cf657eee7f569c73580172209a8f28df761e1ac7a8f43d"} Oct 30 19:37:03.089361 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.089299 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podStartSLOduration=3.089277546 podStartE2EDuration="3.089277546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:03.079492183 +0000 UTC m=+983.993096801" watchObservedRunningTime="2026-10-30 19:37:03.089277546 +0000 UTC m=+984.002882166" Oct 30 19:37:03.249393 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.249354 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:03.604706 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.604618 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:03.828860 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.828434 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:03.828860 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.828496 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:03.854238 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.854194 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:03.888441 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.865958 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:03.936853 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.933253 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:03.936853 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.935494 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:03.970490 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.970452 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:04.025426 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.025385 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:04.025877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:03.990318 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:04.068415 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.025272 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:04.068687 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.068659 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:04.266128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.263455 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:04.303123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.302850 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:37:04.512170 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.510090 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:04.561731 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.561619 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podStartSLOduration=4.561597224 podStartE2EDuration="4.561597224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:03.155012372 +0000 UTC m=+984.068616989" watchObservedRunningTime="2026-10-30 19:37:04.561597224 +0000 UTC m=+985.475201842" Oct 30 19:37:04.763090 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.763047 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:04.792997 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.785555 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:37:04.818666 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.818567 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:04.820044 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.820015 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:37:04.825792 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.825769 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:37:04.843174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.836201 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:04.843174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.836252 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:04.867588 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.867547 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:04.916376 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.916341 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:37:04.937245 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.937197 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:04.955815 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:04.955454 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:37:05.014144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.009634 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:05.014144 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.009693 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:05.031116 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.031050 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.032461 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.031485 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:05.032659 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.032639 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:05.032773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.032349 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:37:05.042702 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.032318 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.044179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.044156 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:37:05.044364 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.044350 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:37:05.076282 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.076179 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.121696 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.121655 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" podUID="2568922e-f36d-41c4-80b5-362fc955a402" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.43:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.636182 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.543211 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.776426 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.776371 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.815137 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.807191 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.828305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.828255 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:37:05.829865 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.829816 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.846654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.845256 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.922186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.917493 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.922186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.917773 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:05.922186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.917814 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:05.956305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.956264 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:05.956583 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.956552 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:05.956758 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.956738 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:05.973264 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.973224 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:05.973543 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:05.973517 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:06.026791 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:06.026757 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:37:06.027389 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:06.027370 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:37:06.028476 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:06.028460 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:37:06.029391 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:06.029376 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:37:06.053125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:06.030129 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:37:06.089132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:06.056927 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:37:06.103128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:06.057210 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:37:06.107124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:06.102250 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:06.113447 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:06.109233 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:37:06.653330 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:06.653290 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:07.117125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.115958 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:37:07.125651 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.125595 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:07.162131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.133326 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:07.187644 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.149354 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:37:07.222856 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.149749 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:37:07.293970 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.220842 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:37:07.356129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.220914 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:37:07.356129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.221069 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:07.356129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.237806 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:07.356129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.343216 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:07.356129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.237872 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:07.356129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.343286 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:07.356129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.332270 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:07.356129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.343333 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:07.356129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.241121 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 19:37:07.357057 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.357012 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 30 19:37:07.463449 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.262209 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:07.702129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.667531 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:07.702129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.667779 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:07.864182 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.855223 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:07.908245 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:07.908193 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:08.105127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.092228 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:08.105127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.092295 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:08.109158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.105962 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:08.109158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.106016 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:08.131182 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.129973 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.143672 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.131769 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.165954 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.165915 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.205154 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.192457 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:37:08.205154 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.192518 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:37:08.205154 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.192550 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:37:08.207813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.207767 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.277756 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.277715 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:37:08.477129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.452600 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:37:08.477129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.452833 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:37:08.477129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.453022 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:08.503237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.478330 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.602256 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.602188 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.674693 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.674649 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.895196 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.894805 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:08.895196 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.894869 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:08.950124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.948708 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:08.950124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.948769 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:08.973130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.969969 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:08.973130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.970029 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:09.005127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:08.997606 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:37:09.134127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.133175 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:09.134127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.133955 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:37:09.191922 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.191876 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:37:09.213639 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.213601 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:09.282513 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.282476 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:09.465181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.449719 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7254-b0200d"] Oct 30 19:37:09.565066 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.565028 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7254-b0200d" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" containerName="test-curl-7254-b0200d" containerID="cri-o://c3c7338fd3ff2a96046bf18fabf9d149f5f8952a3b1fa323bb9db0f33d49901b" gracePeriod=2 Oct 30 19:37:09.565586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.518993 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:37:09.567171 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.519043 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:37:09.567551 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.519064 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:09.567894 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.544190 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" podUID="2568922e-f36d-41c4-80b5-362fc955a402" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.43:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:09.568038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.502255 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:09.589519 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:09.589479 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7254-b0200d\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:37:09.624132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.608294 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:09.624132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.608357 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:09.676028 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.675984 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:09.795286 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.795199 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7254-b0200d"] Oct 30 19:37:09.829138 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.820370 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7254-c84198"] Oct 30 19:37:09.829138 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.821347 4904 cpu_manager.go:414] "containerMap: removing container" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" containerName="test-curl-7254-b0200d" Oct 30 19:37:09.829138 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.821370 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" containerName="test-curl-7254-b0200d" Oct 30 19:37:09.829138 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.821586 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" containerName="test-curl-7254-b0200d" Oct 30 19:37:09.830129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.829886 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:09.830129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.829940 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:09.959128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.930201 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:09.959128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.930262 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:09.970438 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.970406 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7254-c84198" Oct 30 19:37:10.023284 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.991373 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:10.023284 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:09.991481 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:10.023284 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.006739 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:10.023284 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.007141 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:37:10.059435 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.030644 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7f2\" (UniqueName: \"kubernetes.io/projected/2aa8f591-2617-4124-815c-164d53745139-kube-api-access-mk7f2\") pod \"test-curl-7254-c84198\" (UID: \"2aa8f591-2617-4124-815c-164d53745139\") " pod="opendatahub/test-curl-7254-c84198" Oct 30 19:37:10.121328 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.121282 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 19:37:10.154125 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:10.123878 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7254-b0200d\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:37:10.184529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.157121 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:10.184529 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:10.168324 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7254-b0200d\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:37:10.184529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.169339 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:37:10.185693 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.185663 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7f2\" (UniqueName: \"kubernetes.io/projected/2aa8f591-2617-4124-815c-164d53745139-kube-api-access-mk7f2\") pod \"test-curl-7254-c84198\" (UID: \"2aa8f591-2617-4124-815c-164d53745139\") " pod="opendatahub/test-curl-7254-c84198" Oct 30 19:37:10.198134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.194179 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:10.205184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.198551 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:37:10.205184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.198596 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7254-c84198"] Oct 30 19:37:10.275379 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.271262 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:10.289128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.283240 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:10.505127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.501865 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7f2\" (UniqueName: \"kubernetes.io/projected/2aa8f591-2617-4124-815c-164d53745139-kube-api-access-mk7f2\") pod \"test-curl-7254-c84198\" (UID: \"2aa8f591-2617-4124-815c-164d53745139\") " pod="opendatahub/test-curl-7254-c84198" Oct 30 19:37:10.505127 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:10.502272 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7254-b0200d\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:37:10.597140 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.570898 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7254-c84198" Oct 30 19:37:10.597501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.597462 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:10.784220 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.604195 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:10.784220 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.604261 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:37:10.784220 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.604286 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:10.784220 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.736943 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:10.861547 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.848421 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:10.861547 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.848502 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:10.934375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.934345 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:10.934673 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.934630 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:10.960747 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.960714 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:10.961138 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.961032 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:10.992445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:10.992412 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:37:11.007224 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.007186 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:11.053394 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:11.053306 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7254-b0200d\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:37:11.098382 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.098354 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:37:11.292134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.282665 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:37:11.402908 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.306934 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:11.410192 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.308850 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:11.640680 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.334656 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:37:11.691124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.334776 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:37:11.691124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.384730 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 19:37:11.729488 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.729426 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Oct 30 19:37:11.819718 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:11.818617 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7254-b0200d\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:37:11.820980 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.820938 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:11.821915 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.821887 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:11.825460 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.825434 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:11.825622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.825601 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:11.904779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.900957 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:12.029142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:11.971415 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:12.303191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.301252 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:12.303191 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:12.298235 4904 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.005s" Oct 30 19:37:12.303191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.021961 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:12.303191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.302207 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:12.303191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.302435 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:37:12.403752 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.403712 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:12.404363 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.404334 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:12.460056 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.460018 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:12.542159 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.529934 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7254-c84198"] Oct 30 19:37:12.752178 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.733245 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:37:12.752178 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.736045 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:12.848447 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.838626 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:12.848447 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.847548 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:12.848447 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.847603 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:12.884629 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.852614 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:12.884629 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.852868 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:12.921591 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.920245 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:12.976669 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.976625 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:12.976996 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:12.976961 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:13.128179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.071246 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:13.128179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.071311 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:13.128179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.073077 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:13.413152 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.407034 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:13.535543 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.460080 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:13.536621 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.487745 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4"] Oct 30 19:37:13.545387 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.498157 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:13.545820 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.517498 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:37:13.599251 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.596007 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4"] Oct 30 19:37:13.599251 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.596061 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7254-c84198" event={"ID":"2aa8f591-2617-4124-815c-164d53745139","Type":"ContainerStarted","Data":"426fd76186e2259fbd379d8e94377af388247fa0a03ee8bd56ed25bd0f12a00c"} Oct 30 19:37:13.599251 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.596239 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:13.617161 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.616455 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-df776e5c-openshift-default-dockercfg-tbggl\"" Oct 30 19:37:13.692787 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.677191 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 30 19:37:13.781249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.781206 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:13.785760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.781993 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istiod-ca-cert\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:13.785760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.782046 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfsd\" (UniqueName: \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-kube-api-access-lmfsd\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:13.785760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.782074 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-podinfo\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:13.785760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.782152 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-data\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:13.785760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.782183 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-certs\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:13.785760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.782238 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-credential-socket\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:13.785760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.782414 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-token\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:13.785760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.782464 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-envoy\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:13.785760 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.782500 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-socket\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:13.800561 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.800513 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:13.856127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.844215 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:13.856127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.844273 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:13.856127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.844886 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:13.856127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.845690 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:37:13.856127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.853214 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:13.856127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.853865 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:37:13.911128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.885999 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-credential-socket\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.009405 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.009314 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-token\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.022547 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.022514 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-envoy\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.024230 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.024208 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-socket\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.024733 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.024714 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istiod-ca-cert\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.024899 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.024879 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfsd\" (UniqueName: \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-kube-api-access-lmfsd\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.025016 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.025000 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-podinfo\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.025226 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.025205 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-data\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.025367 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.025349 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-certs\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.026198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.025889 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-certs\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.043359 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.886642 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-credential-socket\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.043551 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.027093 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istiod-ca-cert\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.043665 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.925201 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:14.044052 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.024652 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-socket\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.044220 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.954157 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:14.044349 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.044325 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:14.044468 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:13.956689 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:14.044571 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.044554 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:14.045065 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.043194 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-data\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.201788 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.201504 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-token\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.204167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.203996 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-podinfo\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.208406 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.208338 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfsd\" (UniqueName: \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-kube-api-access-lmfsd\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.325736 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.325698 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-envoy\") pod \"e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.360274 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.356520 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:14.527241 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.525633 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:37:14.547180 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.536463 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:37:14.547180 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.536943 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:37:14.549480 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.548659 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:14.549480 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.548933 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:37:14.588807 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.588297 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:14.643571 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.643457 4904 generic.go:356] "Generic (PLEG): container finished" podID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" containerID="c3c7338fd3ff2a96046bf18fabf9d149f5f8952a3b1fa323bb9db0f33d49901b" exitCode=137 Oct 30 19:37:14.825884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.825791 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:37:14.956604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.815076 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:14.956604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.845209 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:14.956604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.954447 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:14.956604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.871642 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:14.956604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.845484 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:14.956604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.933268 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:14.956604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.954984 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:14.956604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.952942 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:14.956604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:14.955046 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:15.019126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.015862 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:15.057133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.052221 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:15.127235 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.123783 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 19:37:15.351941 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.351897 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:37:15.416885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.416860 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4"] Oct 30 19:37:15.445964 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.444251 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:37:15.548752 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.531595 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:15.548752 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.537297 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:15.549357 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.549326 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:15.550073 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.549738 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:15.586791 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.586738 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:15.619533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.619496 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:15.679188 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.617724 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e3e7cd-6367-4413-9969-ee275ab60bfc-kube-api-access-2vq9j\" (UniqueName: \"kubernetes.io/projected/d5e3e7cd-6367-4413-9969-ee275ab60bfc-kube-api-access-2vq9j\") pod \"d5e3e7cd-6367-4413-9969-ee275ab60bfc\" (UID: \"d5e3e7cd-6367-4413-9969-ee275ab60bfc\") " Oct 30 19:37:15.679188 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.588480 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:15.679514 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.679481 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:15.696570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.696537 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e3e7cd-6367-4413-9969-ee275ab60bfc-kube-api-access-2vq9j" pod "d5e3e7cd-6367-4413-9969-ee275ab60bfc" (UID: "d5e3e7cd-6367-4413-9969-ee275ab60bfc"). InnerVolumeSpecName "kube-api-access-2vq9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:15.798459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.798430 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:37:15.846233 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.846205 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:15.904935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.902156 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vq9j\" (UniqueName: \"kubernetes.io/projected/d5e3e7cd-6367-4413-9969-ee275ab60bfc-kube-api-access-2vq9j\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:37:15.988964 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.988931 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:37:15.990479 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.990447 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:37:15.991440 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.991415 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:15.992370 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:15.992340 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:16.005367 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.005345 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:16.025966 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.025932 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:16.026220 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.026192 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:16.047668 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.047639 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:16.047903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.047873 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:16.091276 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.066879 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:16.091276 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.068410 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:16.091276 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.085734 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:37:16.091276 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.087940 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:37:16.091276 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.088288 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:37:16.094123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.091814 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:16.245314 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.225474 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:37:16.447185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.403638 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:16.447185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.446825 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:37:16.636346 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.636313 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:37:16.637652 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.637631 4904 scope.go:122] "RemoveContainer" containerID="c3c7338fd3ff2a96046bf18fabf9d149f5f8952a3b1fa323bb9db0f33d49901b" Oct 30 19:37:16.697649 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.697613 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" event={"ID":"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2","Type":"ContainerStarted","Data":"bb9b054ff7ba0e887406c7bf184e4be42eece503db38d30c3dc34e97f0fc575a"} Oct 30 19:37:16.833764 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.833730 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:16.834031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.833997 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:16.891136 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:16.890162 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7254-b0200d\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" pod="opendatahub/test-curl-7254-b0200d" Oct 30 19:37:16.931711 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.931679 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:16.931924 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.931897 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:16.945480 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.945440 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:16.945691 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.945663 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:16.993295 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.993253 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:16.993819 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:16.993789 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:37:17.137180 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.125983 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:17.162128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.160176 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:17.288905 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.288868 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e3e7cd-6367-4413-9969-ee275ab60bfc" path="/var/lib/kubelet/pods/d5e3e7cd-6367-4413-9969-ee275ab60bfc/volumes" Oct 30 19:37:17.335909 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.335872 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:17.415177 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.413947 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:37:17.415177 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.414318 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:17.636127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.625400 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:37:17.681123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.671427 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:37:17.681123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.678549 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:37:17.681123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.679367 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:37:17.733128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.733050 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7254-c84198" event={"ID":"2aa8f591-2617-4124-815c-164d53745139","Type":"ContainerStarted","Data":"ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf"} Oct 30 19:37:17.742129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.738638 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" event={"ID":"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2","Type":"ContainerStarted","Data":"6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658"} Oct 30 19:37:17.856460 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.856427 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:17.859538 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.859489 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:17.946183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.945173 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:17.946183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.945228 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:17.965189 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.946654 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:17.965189 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:17.946698 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:18.141746 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.112651 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" podStartSLOduration=6.11260852 podStartE2EDuration="6.11260852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:18.087621526 +0000 UTC m=+999.001226143" watchObservedRunningTime="2026-10-30 19:37:18.11260852 +0000 UTC m=+999.026213137" Oct 30 19:37:18.241285 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.232875 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7254-c84198" podStartSLOduration=9.232855758 podStartE2EDuration="9.232855758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:18.144310094 +0000 UTC m=+999.057914711" watchObservedRunningTime="2026-10-30 19:37:18.232855758 +0000 UTC m=+999.146460376" Oct 30 19:37:18.403083 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.402716 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:18.457968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.457937 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:18.751081 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.751053 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:18.752039 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.751995 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:37:18.834128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.829567 4904 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:37:18.834128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.829634 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:37:18.929968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.929915 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:37:18.930491 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.930447 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 19:37:18.951142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.948530 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:37:18.951142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:18.948588 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:37:19.167235 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:19.154217 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7254-c84198"] Oct 30 19:37:19.167235 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:19.162787 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7254-c84198"] Oct 30 19:37:19.810433 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:19.810393 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7254-c84198" podUID="2aa8f591-2617-4124-815c-164d53745139" containerName="test-curl-7254-c84198" containerID="cri-o://ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf" gracePeriod=2 Oct 30 19:37:19.949126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:19.949003 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:19.956129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:19.952401 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:37:20.224589 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:20.224541 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:20.226765 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:20.226740 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:37:20.253095 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:20.253060 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:20.254222 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:20.254205 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:37:22.078711 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:22.078681 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7254-c84198" Oct 30 19:37:22.214250 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:22.214214 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa8f591-2617-4124-815c-164d53745139-kube-api-access-mk7f2\" (UniqueName: \"kubernetes.io/projected/2aa8f591-2617-4124-815c-164d53745139-kube-api-access-mk7f2\") pod \"2aa8f591-2617-4124-815c-164d53745139\" (UID: \"2aa8f591-2617-4124-815c-164d53745139\") " Oct 30 19:37:22.217396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:22.217363 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa8f591-2617-4124-815c-164d53745139-kube-api-access-mk7f2" pod "2aa8f591-2617-4124-815c-164d53745139" (UID: "2aa8f591-2617-4124-815c-164d53745139"). InnerVolumeSpecName "kube-api-access-mk7f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:22.318213 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:22.317322 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk7f2\" (UniqueName: \"kubernetes.io/projected/2aa8f591-2617-4124-815c-164d53745139-kube-api-access-mk7f2\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:37:22.841542 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:22.841504 4904 generic.go:356] "Generic (PLEG): container finished" podID="2aa8f591-2617-4124-815c-164d53745139" containerID="ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf" exitCode=137 Oct 30 19:37:22.841730 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:22.841567 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7254-c84198" Oct 30 19:37:22.841730 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:22.841605 4904 scope.go:122] "RemoveContainer" containerID="ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf" Oct 30 19:37:22.853609 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:22.853587 4904 scope.go:122] "RemoveContainer" containerID="ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf" Oct 30 19:37:22.853946 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:22.853918 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf\": container with ID starting with ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf not found: ID does not exist" containerID="ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf" Oct 30 19:37:22.854037 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:22.853957 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf"} err="failed to get container status \"ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf\": rpc error: code = NotFound desc = could not find container \"ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf\": container with ID starting with ce83b4d5f431bfaddc6e737597bff2aa2213e88a922fc6c8751162084ea5dadf not found: ID does not exist" Oct 30 19:37:22.869063 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:22.869026 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7254-c84198\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="2aa8f591-2617-4124-815c-164d53745139" pod="opendatahub/test-curl-7254-c84198" Oct 30 19:37:23.232056 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:23.232014 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa8f591-2617-4124-815c-164d53745139" path="/var/lib/kubelet/pods/2aa8f591-2617-4124-815c-164d53745139/volumes" Oct 30 19:37:24.516563 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:24.516526 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:37:30.316984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.316953 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj"] Oct 30 19:37:30.317670 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.317654 4904 cpu_manager.go:414] "containerMap: removing container" podUID="2aa8f591-2617-4124-815c-164d53745139" containerName="test-curl-7254-c84198" Oct 30 19:37:30.317732 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.317672 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa8f591-2617-4124-815c-164d53745139" containerName="test-curl-7254-c84198" Oct 30 19:37:30.317798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.317788 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa8f591-2617-4124-815c-164d53745139" containerName="test-curl-7254-c84198" Oct 30 19:37:30.321516 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.321498 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.343781 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.343755 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj"] Oct 30 19:37:30.352509 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.347276 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-maas-api-tls\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.352509 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.347455 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-openshift-service-ca\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.352509 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.347566 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-metrics-certs\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.352509 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.347593 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xvg\" (UniqueName: \"kubernetes.io/projected/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-kube-api-access-95xvg\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.352509 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.349984 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-62b91eb5-serving-cert\"" Oct 30 19:37:30.449533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.449498 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-maas-api-tls\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.449699 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:30.449664 4904 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-serving-cert: secret "maas-api-e2e-ait-62b91eb5-serving-cert" not found Oct 30 19:37:30.449699 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.449679 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-openshift-service-ca\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.449776 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:30.449745 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-maas-api-tls podName:fc1f8d11-da92-4fbc-ae3b-a9052f69aace nodeName:}" failed. No retries permitted until 2026-10-30 19:37:30.949721639 +0000 UTC m=+1011.863326254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-maas-api-tls") pod "maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" (UID: "fc1f8d11-da92-4fbc-ae3b-a9052f69aace") : secret "maas-api-e2e-ait-62b91eb5-serving-cert" not found Oct 30 19:37:30.449865 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.449848 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-metrics-certs\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.449908 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.449883 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xvg\" (UniqueName: \"kubernetes.io/projected/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-kube-api-access-95xvg\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.450456 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.450428 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-openshift-service-ca\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.452622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.452599 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-metrics-certs\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.470816 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.470787 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xvg\" (UniqueName: \"kubernetes.io/projected/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-kube-api-access-95xvg\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.670798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.670714 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg"] Oct 30 19:37:30.676139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.676086 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.698523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.698495 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg"] Oct 30 19:37:30.756038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.754760 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.756038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.754843 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.756038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.754880 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.756038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.754961 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.756038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.755011 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.756038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.755033 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.756038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.755145 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.756038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.755166 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.756038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.755200 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rsq4\" (UniqueName: \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-kube-api-access-6rsq4\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.856606 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.856575 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.856606 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.856611 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.856706 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.856728 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.856827 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsq4\" (UniqueName: \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-kube-api-access-6rsq4\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.856898 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.857023 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857335 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.857057 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857335 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.857133 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857335 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.857266 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857576 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.857554 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857628 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.857562 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.857720 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.857701 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.858289 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.858268 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.860035 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.860015 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.860201 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.860182 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.880565 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.880532 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.881035 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.881014 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rsq4\" (UniqueName: \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-kube-api-access-6rsq4\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:30.958653 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.958618 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-maas-api-tls\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.961277 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.961248 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-maas-api-tls\") pod \"maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:30.987294 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.987269 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx"] Oct 30 19:37:30.992806 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.992787 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:30.995678 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:30.995655 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:31.014249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.014227 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx"] Oct 30 19:37:31.067749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.064654 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.067749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.064725 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.067749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.064775 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.067749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.064838 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.067749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.065050 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.067749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.065180 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aad9d49-47d6-46b0-9d35-258081874d7c-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.067749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.065231 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lln8c\" (UniqueName: \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-kube-api-access-lln8c\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.067749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.065398 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.067749 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.065462 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.207674 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.207762 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.207843 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.207891 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aad9d49-47d6-46b0-9d35-258081874d7c-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.207944 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lln8c\" (UniqueName: \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-kube-api-access-lln8c\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.208052 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.208151 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.208324 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.208395 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.209180 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.209411 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.209634 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.210291 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aad9d49-47d6-46b0-9d35-258081874d7c-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.216173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.211136 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.223902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.219941 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.225974 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.225790 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.240172 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.236995 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lln8c\" (UniqueName: \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-kube-api-access-lln8c\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.241898 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.241870 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:31.288460 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.288429 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.311114 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.311062 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:31.544987 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:37:31.543493 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e21f26d_3a3b_4913_8a9b_0f6801f53456.slice/crio-38ac11e72eb214d6b0b59b2b4c6b32c3334506515d4ae9668e30d8010985daca WatchSource:0}: Error finding container 38ac11e72eb214d6b0b59b2b4c6b32c3334506515d4ae9668e30d8010985daca: Status 404 returned error can't find the container with id 38ac11e72eb214d6b0b59b2b4c6b32c3334506515d4ae9668e30d8010985daca Oct 30 19:37:31.571145 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.565992 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg"] Oct 30 19:37:31.588135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.587300 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:31.588135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.587477 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:31.588135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.587529 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:31.730265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.728605 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj"] Oct 30 19:37:31.757804 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:37:31.757749 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1f8d11_da92_4fbc_ae3b_a9052f69aace.slice/crio-f172a20473e192fc075100c8d1f73a13b7db077eb64ad4954292f79039b5deed WatchSource:0}: Error finding container f172a20473e192fc075100c8d1f73a13b7db077eb64ad4954292f79039b5deed: Status 404 returned error can't find the container with id f172a20473e192fc075100c8d1f73a13b7db077eb64ad4954292f79039b5deed Oct 30 19:37:31.948562 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.947132 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" event={"ID":"6e21f26d-3a3b-4913-8a9b-0f6801f53456","Type":"ContainerStarted","Data":"f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7"} Oct 30 19:37:31.948562 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.947204 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" event={"ID":"6e21f26d-3a3b-4913-8a9b-0f6801f53456","Type":"ContainerStarted","Data":"38ac11e72eb214d6b0b59b2b4c6b32c3334506515d4ae9668e30d8010985daca"} Oct 30 19:37:31.951000 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.950969 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" event={"ID":"fc1f8d11-da92-4fbc-ae3b-a9052f69aace","Type":"ContainerStarted","Data":"f172a20473e192fc075100c8d1f73a13b7db077eb64ad4954292f79039b5deed"} Oct 30 19:37:31.996271 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.996186 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:31.997951 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:31.997890 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podStartSLOduration=1.997872595 podStartE2EDuration="1.997872595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:31.993464557 +0000 UTC m=+1012.907069175" watchObservedRunningTime="2026-10-30 19:37:31.997872595 +0000 UTC m=+1012.911477212" Oct 30 19:37:32.016036 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:32.016007 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx"] Oct 30 19:37:32.021394 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:32.021338 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:32.021512 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:32.021448 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:32.021512 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:32.021496 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:32.997529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:32.997497 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:32.998088 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:32.998057 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:33.007812 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.007781 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" event={"ID":"fc1f8d11-da92-4fbc-ae3b-a9052f69aace","Type":"ContainerStarted","Data":"ebe11c529e9366c405ba7e8e9fc30b661d40c3082aaea28c703bf422ffa18333"} Oct 30 19:37:33.009367 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.009345 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:33.012292 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.012268 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" event={"ID":"7aad9d49-47d6-46b0-9d35-258081874d7c","Type":"ContainerStarted","Data":"47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c"} Oct 30 19:37:33.012508 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.012487 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" event={"ID":"7aad9d49-47d6-46b0-9d35-258081874d7c","Type":"ContainerStarted","Data":"68cf108198b62268019ee8324731250959ac933641e26de4cba7f246d9d337b4"} Oct 30 19:37:33.082038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.081983 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" podStartSLOduration=2.589413274 podStartE2EDuration="3.081967977s" totalImagesPullingTime="492.554703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:33.079480387 +0000 UTC m=+1013.993085005" watchObservedRunningTime="2026-10-30 19:37:33.081967977 +0000 UTC m=+1013.995572593" Oct 30 19:37:33.138386 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.138322 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podStartSLOduration=3.138300308 podStartE2EDuration="3.138300308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:33.130413904 +0000 UTC m=+1014.044018521" watchObservedRunningTime="2026-10-30 19:37:33.138300308 +0000 UTC m=+1014.051904926" Oct 30 19:37:33.316349 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.315939 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:33.316643 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.316595 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:33.321644 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.321622 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:33.748129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.747493 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d"] Oct 30 19:37:33.782855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.764619 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d"] Oct 30 19:37:33.782855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.764734 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:33.792664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.787657 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5016f41a-serving-cert\"" Oct 30 19:37:33.823499 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.821182 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-metrics-certs\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:33.823499 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.821427 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5747a0b5-a568-458c-b409-242575beeca4-openshift-service-ca\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:33.823499 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.821492 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-maas-api-tls\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:33.823499 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.821695 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46vx\" (UniqueName: \"kubernetes.io/projected/5747a0b5-a568-458c-b409-242575beeca4-kube-api-access-m46vx\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:33.984131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.962922 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46vx\" (UniqueName: \"kubernetes.io/projected/5747a0b5-a568-458c-b409-242575beeca4-kube-api-access-m46vx\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:33.984131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.963017 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-metrics-certs\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:33.984131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.963268 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5747a0b5-a568-458c-b409-242575beeca4-openshift-service-ca\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:33.984131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.963324 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-maas-api-tls\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:34.003517 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:33.992059 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5747a0b5-a568-458c-b409-242575beeca4-openshift-service-ca\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:34.045549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:34.034907 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:34.045549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:34.034965 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:34.069198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:34.051914 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-maas-api-tls\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:34.240124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:34.230701 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-metrics-certs\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:34.369579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:34.368827 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46vx\" (UniqueName: \"kubernetes.io/projected/5747a0b5-a568-458c-b409-242575beeca4-kube-api-access-m46vx\") pod \"maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:34.394409 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:34.387524 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:34.394409 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:34.387634 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:34.580714 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:34.580622 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:34.995673 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:34.995627 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:35.025925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.025882 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:35.026366 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.025944 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:35.057188 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.056220 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:35.313413 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.313087 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:35.313867 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.313812 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:35.493129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.480642 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:35.578650 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.578553 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:35.637129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.635364 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx"] Oct 30 19:37:35.689332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.689292 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx"] Oct 30 19:37:35.703225 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.695581 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:35.781125 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:37:35.764705 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5747a0b5_a568_458c_b409_242575beeca4.slice/crio-ad670e6ff29e76266bb45bcf69e38352e30d368a6335650ec1e942a5191bf5a7 WatchSource:0}: Error finding container ad670e6ff29e76266bb45bcf69e38352e30d368a6335650ec1e942a5191bf5a7: Status 404 returned error can't find the container with id ad670e6ff29e76266bb45bcf69e38352e30d368a6335650ec1e942a5191bf5a7 Oct 30 19:37:35.802125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.797373 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:37:35.802125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.797439 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:37:35.802125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.797600 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6vscc\"" Oct 30 19:37:35.802125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.797655 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:37:35.802125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.800311 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:35.868779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.863820 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:35.868779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.863874 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1162a95f-a50d-4cc6-b0f7-835385f46757-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:35.868779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.863936 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:35.868779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.863971 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:35.868779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.864023 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7d4\" (UniqueName: \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-kube-api-access-gf7d4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:35.868779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.864155 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:35.868779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.864228 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.959166 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d"] Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.969620 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.969710 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.969922 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.969952 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1162a95f-a50d-4cc6-b0f7-835385f46757-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.970014 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.970044 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.970094 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7d4\" (UniqueName: \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-kube-api-access-gf7d4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:35.971326 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.032843 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:36.053560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.032895 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:36.162995 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.145983 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7d4\" (UniqueName: \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-kube-api-access-gf7d4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.251881 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.251803 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.252458 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.252410 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.258123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.252743 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1162a95f-a50d-4cc6-b0f7-835385f46757-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.259314 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.258600 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.313127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.302991 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zf9jx\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:36.332138 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.330836 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:36.332138 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.330888 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:36.340126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.339174 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:36.376126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.372991 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" event={"ID":"5747a0b5-a568-458c-b409-242575beeca4","Type":"ContainerStarted","Data":"ad670e6ff29e76266bb45bcf69e38352e30d368a6335650ec1e942a5191bf5a7"} Oct 30 19:37:36.473409 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:36.473380 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:37.003130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:37.002660 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:37.003130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:37.002718 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:37.321983 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:37.316493 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:37.321983 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:37.316554 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:37.400471 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:37.400438 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx"] Oct 30 19:37:37.489056 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:37:37.489025 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1162a95f_a50d_4cc6_b0f7_835385f46757.slice/crio-832080d7f99a978bbb9f5b4651251145b86e2ade037c8e55f9123e7439a898a0 WatchSource:0}: Error finding container 832080d7f99a978bbb9f5b4651251145b86e2ade037c8e55f9123e7439a898a0: Status 404 returned error can't find the container with id 832080d7f99a978bbb9f5b4651251145b86e2ade037c8e55f9123e7439a898a0 Oct 30 19:37:37.537763 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:37.536591 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" event={"ID":"5747a0b5-a568-458c-b409-242575beeca4","Type":"ContainerStarted","Data":"6cce0742be52367fbd4155db61a8b5e72445a9f2f0b4a8896baf24c0312f97c1"} Oct 30 19:37:37.537763 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:37.537712 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:38.000846 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:38.000372 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:38.000846 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:38.000449 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:38.337183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:38.322132 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:38.337183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:38.322202 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:39.069129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:39.064768 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:39.069129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:39.064826 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:39.190122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:39.150020 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" event={"ID":"1162a95f-a50d-4cc6-b0f7-835385f46757","Type":"ContainerStarted","Data":"832080d7f99a978bbb9f5b4651251145b86e2ade037c8e55f9123e7439a898a0"} Oct 30 19:37:39.627130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:39.613721 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:39.627130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:39.613787 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:39.816558 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:39.816507 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:39.976285 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:39.976247 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:37:39.997698 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:39.997667 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:39.997993 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:39.997952 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:40.057739 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.057656 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:40.114430 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.114391 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:40.178778 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.178739 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:40.296602 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.296508 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:40.349139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.349087 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:40.349392 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.349367 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:40.412007 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.411978 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:37:40.413496 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.413463 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:37:40.497257 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.465834 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:40.949853 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.540172 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:40.951002 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.669756 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:37:40.981477 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.727253 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:40.982981 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.798184 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:37:40.983405 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.818283 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:40.984859 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.891397 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" podStartSLOduration=7.240379726 podStartE2EDuration="7.891374754s" totalImagesPullingTime="650.995028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:37.607899383 +0000 UTC m=+1018.521504003" watchObservedRunningTime="2026-10-30 19:37:40.891374754 +0000 UTC m=+1021.804979373" Oct 30 19:37:40.994125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.985985 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:37:41.013942 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:40.986788 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:41.015566 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.015530 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:37:41.049358 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.049328 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:41.049644 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.049599 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:41.071447 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.071391 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:41.074091 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.074072 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:37:41.175184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.171188 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:41.188136 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.188059 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:41.191602 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.191583 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:37:41.331224 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.297991 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:41.331837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.331757 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:41.332080 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.332032 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:41.433197 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.433082 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:41.678140 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.675567 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:41.716407 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.716265 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 30 19:37:41.963869 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.963826 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:37:41.964886 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.964865 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:37:42.031533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.988770 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 19:37:42.031533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.989187 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:42.031533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.989469 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 19:37:42.031533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.990128 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:37:42.031533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.990182 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:37:42.031533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:41.990220 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:37:42.031533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.017174 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:42.031533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.031071 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:42.031533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.031141 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:42.032128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.031839 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:37:42.032555 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.032523 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:42.039133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.032726 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:37:42.039133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.038482 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:37:42.039133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.038798 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:37:42.103673 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.103232 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:42.195472 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.195236 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:37:42.300884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.300778 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:42.407965 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.407929 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:42.475125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.446686 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:43.166390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.444438 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:37:43.166390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:42.708449 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 30 19:37:43.166390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.009621 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 19:37:43.166390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.037041 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:43.166390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.089669 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:43.166390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.112144 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:43.166390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.136507 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:43.166390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.140539 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:37:43.166390 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.141757 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:37:43.226133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.221143 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:43.287705 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.224413 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:37:43.411565 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.411527 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:43.412464 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.412428 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:43.609937 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.609228 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:43.609937 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.609303 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:43.669183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.665433 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:37:43.723324 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.723264 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:43.834979 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.834933 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:44.002358 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.997714 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:37:44.002358 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:43.997773 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:37:44.015566 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.015523 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:44.043560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.042787 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:44.167129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.123391 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:44.274904 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.145759 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:44.275639 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.146697 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:44.288127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.146724 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:37:44.288480 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.288439 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:44.288863 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.146743 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:37:44.289789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.289765 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:37:44.342129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.338798 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:44.342129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.338858 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:44.606544 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.606420 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:44.606824 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.606775 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:44.707130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:44.680498 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:37:45.085333 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.085284 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:37:45.098129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.094428 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:45.112133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.099737 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:45.125452 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.106614 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:45.296139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.280007 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:45.296139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.280885 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:45.297338 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.297308 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:45.298193 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.298159 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:45.343146 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.298351 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:45.354445 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.313655 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:45.355412 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.355379 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:45.355616 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.320692 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:37:45.356117 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.320631 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:37:45.446285 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.444170 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:37:45.855158 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:45.852246 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:46.009527 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.009493 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:46.098506 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.098424 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:46.099004 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.098858 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:46.148995 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.147208 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:46.148995 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.147278 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:46.210821 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.210779 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 19:37:46.285665 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.285625 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:37:46.286406 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.286378 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:46.299966 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.299930 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:46.331236 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.331206 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:46.332808 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.332780 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:37:46.333019 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.332992 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 19:37:46.359613 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.359566 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:46.360232 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.360203 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:37:46.364850 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.364823 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:37:46.365274 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.365248 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:46.701132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.700661 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:37:46.733613 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.732674 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:37:46.854251 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:46.854206 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:47.237324 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.212207 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:47.456149 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.273358 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 19:37:47.456149 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.281354 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:47.474611 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.374381 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:37:47.491211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.426474 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 30 19:37:47.507899 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.491741 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:47.689273 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.492758 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:37:47.690006 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.508966 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:47.690647 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.525770 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 19:37:47.690841 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.604327 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:47.690987 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.690960 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:47.897182 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.883479 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:47.948141 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:47.947806 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:48.131235 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.131200 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:37:48.589780 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.589717 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Oct 30 19:37:48.591572 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.591553 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:37:48.592451 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.592424 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:48.618553 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.618513 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:48.648007 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.624280 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:48.648007 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.624616 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 19:37:48.745312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.652545 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x4dsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:48.745312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.652603 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" podUID="2c012900-326d-42d5-8076-3bbbdbb1e852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:48.745312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.655349 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:48.745312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.655399 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:48.745312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.708808 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:48.745312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.739286 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:48.749799 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.749749 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 19:37:48.797129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.791572 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:48.816880 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.816828 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:48.913401 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.913309 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:37:48.990830 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.957621 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ggdsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:49.019962 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.019920 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" podUID="0c8a2a4b-6240-40fa-a873-d2300e317a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:49.074272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.032226 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:49.088194 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:48.957676 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:49.578129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.577033 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:37:49.604128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.591888 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:49.604128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.592766 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:49.604128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.592940 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:49.625576 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.624344 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:49.642586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.626004 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:37:49.642586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.627375 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:49.642586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.636732 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:37:49.642586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.637347 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:49.688093 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.682537 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79x29" podUID="2568922e-f36d-41c4-80b5-362fc955a402" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.43:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:49.707464 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.707419 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:49.707711 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.707691 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:49.709505 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.709468 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:49.715575 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.715550 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:37:49.747221 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:49.746272 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:50.014687 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.014644 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:50.039244 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.033197 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8k4b6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:50.039244 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.033256 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" podUID="8da395a9-9be0-46ba-b397-ee54354bf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:50.096155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.085762 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:50.096155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.086321 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:37:50.138741 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.097904 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:37:50.325047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.317782 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:50.582470 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.581535 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:37:50.584451 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.583282 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:50.584451 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.583708 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:37:50.593205 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.593173 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 19:37:50.594320 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.594293 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:50.601132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.594840 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:37:50.651300 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.651259 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:50.651779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.651571 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:50.726497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.726461 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:50.948284 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.756489 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:50.948284 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.892945 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:37:50.948284 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.946198 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:50.948284 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:50.947195 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:51.032588 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.016250 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:37:51.173129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.095496 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:51.248763 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.107574 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:51.250416 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.108976 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:51.251413 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.145586 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:51.543645 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.529405 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:37:51.589931 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.583566 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:51.589931 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.584624 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:51.634488 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.634443 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:51.729246 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.729185 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:51.833966 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.797199 4904 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:51.833966 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.823320 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 19:37:51.833966 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.823969 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 19:37:51.958129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:51.951417 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:52.001943 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:52.001916 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:37:52.034155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:52.029214 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:52.038279 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:52.034782 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:37:52.058621 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:52.058596 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:37:52.088683 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:52.088609 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:37:52.142290 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:52.142266 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:37:52.193082 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:52.193055 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:37:52.231125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:52.211711 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:37:52.293131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:52.291391 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:37:54.520469 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:37:54.520433 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:37:55.121673 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:55.121636 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:55.121808 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:55.121719 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:37:55.621331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:55.621299 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" event={"ID":"1162a95f-a50d-4cc6-b0f7-835385f46757","Type":"ContainerStarted","Data":"73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad"} Oct 30 19:37:55.621750 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:55.621385 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:37:55.709007 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:55.708937 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" podStartSLOduration=3.091290704 podStartE2EDuration="20.708917052s" totalImagesPullingTime="17.617626348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:55.704726933 +0000 UTC m=+1036.618331558" watchObservedRunningTime="2026-10-30 19:37:55.708917052 +0000 UTC m=+1036.622521677" Oct 30 19:37:56.632933 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:37:56.632905 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:38:11.025486 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:11.025404 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:11.111127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:11.105203 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:11.130128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:11.123182 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:11.231145 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:11.231073 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:11.304798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:11.303775 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:19.801270 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:19.801230 4904 patch_prober.go:29] interesting pod/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:38:19.801676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:19.801292 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:21.312790 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:21.312753 4904 generic.go:356] "Generic (PLEG): container finished" podID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerID="6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658" exitCode=0 Oct 30 19:38:21.313355 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:21.312804 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" event={"ID":"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2","Type":"ContainerDied","Data":"6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658"} Oct 30 19:38:21.313432 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:21.313372 4904 scope.go:122] "RemoveContainer" containerID="6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658" Oct 30 19:38:21.315167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:21.315130 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:38:21.315272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:21.315214 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:38:21.315272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:21.315249 4904 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:38:22.354220 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:22.354184 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" event={"ID":"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2","Type":"ContainerStarted","Data":"ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96"} Oct 30 19:38:22.397329 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:22.397296 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:38:22.397611 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:22.397585 4904 patch_prober.go:29] interesting pod/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:38:22.397716 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:22.397639 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:38:23.080590 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:23.080557 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj"] Oct 30 19:38:23.080794 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:23.080766 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" podUID="fc1f8d11-da92-4fbc-ae3b-a9052f69aace" containerName="maas-api" containerID="cri-o://ebe11c529e9366c405ba7e8e9fc30b661d40c3082aaea28c703bf422ffa18333" gracePeriod=30 Oct 30 19:38:23.408186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:23.406632 4904 patch_prober.go:29] interesting pod/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:38:23.408186 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:23.406689 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:38:23.608519 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:23.608388 4904 generic.go:356] "Generic (PLEG): container finished" podID="fc1f8d11-da92-4fbc-ae3b-a9052f69aace" containerID="ebe11c529e9366c405ba7e8e9fc30b661d40c3082aaea28c703bf422ffa18333" exitCode=0 Oct 30 19:38:23.608885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:23.608444 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" event={"ID":"fc1f8d11-da92-4fbc-ae3b-a9052f69aace","Type":"ContainerDied","Data":"ebe11c529e9366c405ba7e8e9fc30b661d40c3082aaea28c703bf422ffa18333"} Oct 30 19:38:24.236154 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.214324 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:38:24.277435 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.277390 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-kube-api-access-95xvg\" (UniqueName: \"kubernetes.io/projected/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-kube-api-access-95xvg\") pod \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " Oct 30 19:38:24.277714 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.277700 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-metrics-certs\") pod \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " Oct 30 19:38:24.277812 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.277803 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-openshift-service-ca\") pod \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " Oct 30 19:38:24.277924 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.277915 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-maas-api-tls\") pod \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\" (UID: \"fc1f8d11-da92-4fbc-ae3b-a9052f69aace\") " Oct 30 19:38:24.279079 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.279051 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-openshift-service-ca" pod "fc1f8d11-da92-4fbc-ae3b-a9052f69aace" (UID: "fc1f8d11-da92-4fbc-ae3b-a9052f69aace"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:24.283359 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.283335 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-maas-api-tls" pod "fc1f8d11-da92-4fbc-ae3b-a9052f69aace" (UID: "fc1f8d11-da92-4fbc-ae3b-a9052f69aace"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:24.283620 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.283598 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-kube-api-access-95xvg" pod "fc1f8d11-da92-4fbc-ae3b-a9052f69aace" (UID: "fc1f8d11-da92-4fbc-ae3b-a9052f69aace"). InnerVolumeSpecName "kube-api-access-95xvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:24.284202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.284185 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-metrics-certs" pod "fc1f8d11-da92-4fbc-ae3b-a9052f69aace" (UID: "fc1f8d11-da92-4fbc-ae3b-a9052f69aace"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:24.389461 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.389427 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:24.389461 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.389459 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:24.389739 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.389471 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:24.389739 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.389484 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95xvg\" (UniqueName: \"kubernetes.io/projected/fc1f8d11-da92-4fbc-ae3b-a9052f69aace-kube-api-access-95xvg\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:24.398972 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.398933 4904 patch_prober.go:29] interesting pod/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:38:24.399253 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.399222 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:38:24.732126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.727047 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" event={"ID":"fc1f8d11-da92-4fbc-ae3b-a9052f69aace","Type":"ContainerDied","Data":"f172a20473e192fc075100c8d1f73a13b7db077eb64ad4954292f79039b5deed"} Oct 30 19:38:24.732126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.727158 4904 scope.go:122] "RemoveContainer" containerID="ebe11c529e9366c405ba7e8e9fc30b661d40c3082aaea28c703bf422ffa18333" Oct 30 19:38:24.732126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.727398 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj" Oct 30 19:38:24.784130 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:38:24.778537 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:38:24.888135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.886032 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj"] Oct 30 19:38:24.888135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:24.887510 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62b91eb5-6fcfd4cf99-ncqxj"] Oct 30 19:38:25.260586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:25.260541 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1f8d11-da92-4fbc-ae3b-a9052f69aace" path="/var/lib/kubelet/pods/fc1f8d11-da92-4fbc-ae3b-a9052f69aace/volumes" Oct 30 19:38:25.397255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:25.397218 4904 patch_prober.go:29] interesting pod/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:38:25.397438 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:25.397274 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:38:26.402126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:26.397235 4904 patch_prober.go:29] interesting pod/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:38:26.402126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:26.397299 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:38:27.282264 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:27.282235 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d"] Oct 30 19:38:27.283308 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:27.283070 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" podUID="5747a0b5-a568-458c-b409-242575beeca4" containerName="maas-api" containerID="cri-o://6cce0742be52367fbd4155db61a8b5e72445a9f2f0b4a8896baf24c0312f97c1" gracePeriod=30 Oct 30 19:38:27.397463 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:27.397425 4904 patch_prober.go:29] interesting pod/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:38:27.397658 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:27.397489 4904 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:38:27.813973 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:27.813936 4904 generic.go:356] "Generic (PLEG): container finished" podID="5747a0b5-a568-458c-b409-242575beeca4" containerID="6cce0742be52367fbd4155db61a8b5e72445a9f2f0b4a8896baf24c0312f97c1" exitCode=0 Oct 30 19:38:27.847564 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:27.847526 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" event={"ID":"5747a0b5-a568-458c-b409-242575beeca4","Type":"ContainerDied","Data":"6cce0742be52367fbd4155db61a8b5e72445a9f2f0b4a8896baf24c0312f97c1"} Oct 30 19:38:28.000015 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:27.999954 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:38:28.054782 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.054746 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-maas-api-tls\") pod \"5747a0b5-a568-458c-b409-242575beeca4\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " Oct 30 19:38:28.054984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.054827 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-metrics-certs\") pod \"5747a0b5-a568-458c-b409-242575beeca4\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " Oct 30 19:38:28.054984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.054883 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5747a0b5-a568-458c-b409-242575beeca4-kube-api-access-m46vx\" (UniqueName: \"kubernetes.io/projected/5747a0b5-a568-458c-b409-242575beeca4-kube-api-access-m46vx\") pod \"5747a0b5-a568-458c-b409-242575beeca4\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " Oct 30 19:38:28.055156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.054988 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5747a0b5-a568-458c-b409-242575beeca4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5747a0b5-a568-458c-b409-242575beeca4-openshift-service-ca\") pod \"5747a0b5-a568-458c-b409-242575beeca4\" (UID: \"5747a0b5-a568-458c-b409-242575beeca4\") " Oct 30 19:38:28.056198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.056064 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5747a0b5-a568-458c-b409-242575beeca4-openshift-service-ca" pod "5747a0b5-a568-458c-b409-242575beeca4" (UID: "5747a0b5-a568-458c-b409-242575beeca4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:28.061602 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.061554 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-maas-api-tls" pod "5747a0b5-a568-458c-b409-242575beeca4" (UID: "5747a0b5-a568-458c-b409-242575beeca4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:28.061822 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.061802 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-metrics-certs" pod "5747a0b5-a568-458c-b409-242575beeca4" (UID: "5747a0b5-a568-458c-b409-242575beeca4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:28.069393 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.069288 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5747a0b5-a568-458c-b409-242575beeca4-kube-api-access-m46vx" pod "5747a0b5-a568-458c-b409-242575beeca4" (UID: "5747a0b5-a568-458c-b409-242575beeca4"). InnerVolumeSpecName "kube-api-access-m46vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:28.158668 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.158634 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.158885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.158871 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5747a0b5-a568-458c-b409-242575beeca4-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.158974 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.158963 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m46vx\" (UniqueName: \"kubernetes.io/projected/5747a0b5-a568-458c-b409-242575beeca4-kube-api-access-m46vx\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.159065 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.159051 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5747a0b5-a568-458c-b409-242575beeca4-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.530393 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.529402 4904 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:38:28.531079 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.531053 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:38:28.532051 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.532011 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:38:28.859513 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.859413 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" event={"ID":"5747a0b5-a568-458c-b409-242575beeca4","Type":"ContainerDied","Data":"ad670e6ff29e76266bb45bcf69e38352e30d368a6335650ec1e942a5191bf5a7"} Oct 30 19:38:28.859513 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.859423 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d" Oct 30 19:38:28.860039 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.859520 4904 scope.go:122] "RemoveContainer" containerID="6cce0742be52367fbd4155db61a8b5e72445a9f2f0b4a8896baf24c0312f97c1" Oct 30 19:38:28.917953 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.917904 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d"] Oct 30 19:38:28.922799 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:28.922767 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5016f41a-8f579dff6-qw45d"] Oct 30 19:38:29.225058 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:29.225025 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5747a0b5-a568-458c-b409-242575beeca4" path="/var/lib/kubelet/pods/5747a0b5-a568-458c-b409-242575beeca4/volumes" Oct 30 19:38:32.052124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:32.051210 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4"] Oct 30 19:38:32.052124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:32.051518 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" containerID="cri-o://ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96" gracePeriod=30 Oct 30 19:38:37.336825 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.336765 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:38:37.466419 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466386 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-credential-socket\") pod \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " Oct 30 19:38:37.466603 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466472 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-token\" (UniqueName: \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-token\") pod \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " Oct 30 19:38:37.466603 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466533 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-podinfo\") pod \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " Oct 30 19:38:37.466721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466600 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istiod-ca-cert\") pod \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " Oct 30 19:38:37.466721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466632 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-envoy\") pod \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " Oct 30 19:38:37.466721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466670 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-certs\") pod \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " Oct 30 19:38:37.466721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466682 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-credential-socket" pod "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" (UID: "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:37.466917 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466754 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-kube-api-access-lmfsd\" (UniqueName: \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-kube-api-access-lmfsd\") pod \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " Oct 30 19:38:37.466917 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466826 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-data\") pod \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " Oct 30 19:38:37.467013 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466936 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-socket\") pod \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\" (UID: \"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2\") " Oct 30 19:38:37.467532 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.466962 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istiod-ca-cert" pod "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" (UID: "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:37.467532 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.467317 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-certs" pod "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" (UID: "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:37.467532 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.467331 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-data" pod "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" (UID: "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:37.467532 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.467502 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-socket" pod "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" (UID: "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:37.467817 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.467798 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:37.467877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.467818 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:37.467877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.467832 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:37.467877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.467848 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:37.467877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.467861 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:37.469835 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.469807 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-token" pod "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" (UID: "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:37.470183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.470151 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-kube-api-access-lmfsd" pod "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" (UID: "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2"). InnerVolumeSpecName "kube-api-access-lmfsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:37.470294 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.470220 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-podinfo" pod "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" (UID: "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:37.470294 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.470274 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-envoy" pod "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" (UID: "4d292dcb-ffb3-4398-abc5-2b74ac80c8f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:37.569806 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.569776 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:37.569806 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.569803 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:37.569806 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.569812 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:37.570159 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.569821 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmfsd\" (UniqueName: \"kubernetes.io/projected/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2-kube-api-access-lmfsd\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:38:37.961919 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.961879 4904 generic.go:356] "Generic (PLEG): container finished" podID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerID="ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96" exitCode=0 Oct 30 19:38:37.962120 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.961917 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" event={"ID":"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2","Type":"ContainerDied","Data":"ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96"} Oct 30 19:38:37.962120 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.961941 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" Oct 30 19:38:37.962120 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.961978 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4" event={"ID":"4d292dcb-ffb3-4398-abc5-2b74ac80c8f2","Type":"ContainerDied","Data":"bb9b054ff7ba0e887406c7bf184e4be42eece503db38d30c3dc34e97f0fc575a"} Oct 30 19:38:37.962120 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.962002 4904 scope.go:122] "RemoveContainer" containerID="ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96" Oct 30 19:38:37.978376 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.978352 4904 scope.go:122] "RemoveContainer" containerID="6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658" Oct 30 19:38:37.988987 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.988969 4904 scope.go:122] "RemoveContainer" containerID="ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96" Oct 30 19:38:37.989392 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:38:37.989373 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96\": container with ID starting with ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96 not found: ID does not exist" containerID="ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96" Oct 30 19:38:37.989452 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.989403 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96"} err="failed to get container status \"ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96\": rpc error: code = NotFound desc = could not find container \"ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96\": container with ID starting with ab19736ed6b957034bc7d18bbab88dac3c9b8c8dcae2cbe8516254b1cba5af96 not found: ID does not exist" Oct 30 19:38:37.989452 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.989421 4904 scope.go:122] "RemoveContainer" containerID="6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658" Oct 30 19:38:37.989693 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:38:37.989673 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658\": container with ID starting with 6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658 not found: ID does not exist" containerID="6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658" Oct 30 19:38:37.989744 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:37.989708 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658"} err="failed to get container status \"6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658\": rpc error: code = NotFound desc = could not find container \"6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658\": container with ID starting with 6695a83e66dd513a157f06bc3521ef3b3d6f318f2153f2bba0e4429ce59fb658 not found: ID does not exist" Oct 30 19:38:38.040809 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:38.040762 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4"] Oct 30 19:38:38.050964 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:38.050943 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-df776e5c-openshift-default-55cbc6f865-wwnq4"] Oct 30 19:38:39.236661 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:38:39.236627 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" path="/var/lib/kubelet/pods/4d292dcb-ffb3-4398-abc5-2b74ac80c8f2/volumes" Oct 30 19:38:54.533485 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:38:54.533438 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:39:24.514232 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:39:24.514147 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:39:54.521593 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:39:54.521545 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:40:24.513076 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:40:24.513042 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:40:25.624452 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.624415 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v"] Oct 30 19:40:25.625389 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625371 4904 cpu_manager.go:414] "containerMap: removing container" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" Oct 30 19:40:25.625452 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625392 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" Oct 30 19:40:25.625452 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625442 4904 cpu_manager.go:414] "containerMap: removing container" podUID="fc1f8d11-da92-4fbc-ae3b-a9052f69aace" containerName="maas-api" Oct 30 19:40:25.625523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625452 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1f8d11-da92-4fbc-ae3b-a9052f69aace" containerName="maas-api" Oct 30 19:40:25.625523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625486 4904 cpu_manager.go:414] "containerMap: removing container" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" Oct 30 19:40:25.625523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625494 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" Oct 30 19:40:25.625523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625515 4904 cpu_manager.go:414] "containerMap: removing container" podUID="5747a0b5-a568-458c-b409-242575beeca4" containerName="maas-api" Oct 30 19:40:25.625523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625522 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5747a0b5-a568-458c-b409-242575beeca4" containerName="maas-api" Oct 30 19:40:25.625703 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625670 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="5747a0b5-a568-458c-b409-242575beeca4" containerName="maas-api" Oct 30 19:40:25.625703 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625689 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" Oct 30 19:40:25.625769 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.625704 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1f8d11-da92-4fbc-ae3b-a9052f69aace" containerName="maas-api" Oct 30 19:40:25.630082 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.630061 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.648799 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.648774 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-e9de60-serving-cert\"" Oct 30 19:40:25.650039 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.650007 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v"] Oct 30 19:40:25.658053 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.658008 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174adb82-e0b5-488c-bce2-91d299a6afa9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.658523 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.658480 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vzc\" (UniqueName: \"kubernetes.io/projected/174adb82-e0b5-488c-bce2-91d299a6afa9-kube-api-access-k7vzc\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.659040 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.659006 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.659347 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.659328 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.761396 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.761359 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.761629 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.761460 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174adb82-e0b5-488c-bce2-91d299a6afa9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.761629 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.761562 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vzc\" (UniqueName: \"kubernetes.io/projected/174adb82-e0b5-488c-bce2-91d299a6afa9-kube-api-access-k7vzc\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.761787 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.761751 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.762439 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.762412 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174adb82-e0b5-488c-bce2-91d299a6afa9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.764119 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.764084 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.764426 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.764409 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.782641 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.782606 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vzc\" (UniqueName: \"kubernetes.io/projected/174adb82-e0b5-488c-bce2-91d299a6afa9-kube-api-access-k7vzc\") pod \"maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:25.947539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:25.947505 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:26.275862 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:26.275834 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v"] Oct 30 19:40:26.276887 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:40:26.276860 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174adb82_e0b5_488c_bce2_91d299a6afa9.slice/crio-cc4a4a1c07f083dd4d9f00cf6d36c45992e8908f4f799def3ed657c2fc49cec1 WatchSource:0}: Error finding container cc4a4a1c07f083dd4d9f00cf6d36c45992e8908f4f799def3ed657c2fc49cec1: Status 404 returned error can't find the container with id cc4a4a1c07f083dd4d9f00cf6d36c45992e8908f4f799def3ed657c2fc49cec1 Oct 30 19:40:26.437198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:26.435431 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" event={"ID":"174adb82-e0b5-488c-bce2-91d299a6afa9","Type":"ContainerStarted","Data":"cc4a4a1c07f083dd4d9f00cf6d36c45992e8908f4f799def3ed657c2fc49cec1"} Oct 30 19:40:27.453475 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:27.453437 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" event={"ID":"174adb82-e0b5-488c-bce2-91d299a6afa9","Type":"ContainerStarted","Data":"5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d"} Oct 30 19:40:27.453975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:27.453514 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:27.498845 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:27.498783 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" podStartSLOduration=2.016451283 podStartE2EDuration="2.49876376s" totalImagesPullingTime="482.312477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:27.496484312 +0000 UTC m=+1188.410088930" watchObservedRunningTime="2026-10-30 19:40:27.49876376 +0000 UTC m=+1188.412368379" Oct 30 19:40:33.468252 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:33.468222 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:40:39.480051 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:39.480016 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:40:39.482257 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:39.482234 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:40:39.503351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:39.503327 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:40:39.505533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:40:39.505514 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:40:54.534907 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:40:54.534812 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:41:03.267311 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:03.267259 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x4dsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:41:03.270896 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:03.268052 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" podUID="2c012900-326d-42d5-8076-3bbbdbb1e852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:41:24.409788 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.409758 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq"] Oct 30 19:41:24.410545 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.410530 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d292dcb-ffb3-4398-abc5-2b74ac80c8f2" containerName="istio-proxy" Oct 30 19:41:24.413778 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.413762 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.442556 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.442526 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58h2n\" (UniqueName: \"kubernetes.io/projected/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-kube-api-access-58h2n\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.442826 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.442796 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-openshift-service-ca\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.443010 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.442984 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-metrics-certs\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.443135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.443091 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-maas-api-tls\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.445619 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.445592 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-67953efd-serving-cert\"" Oct 30 19:41:24.445882 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.445858 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq"] Oct 30 19:41:24.515132 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:41:24.515076 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:41:24.545006 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.544975 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-metrics-certs\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.545210 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.545116 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-maas-api-tls\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.545289 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.545237 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58h2n\" (UniqueName: \"kubernetes.io/projected/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-kube-api-access-58h2n\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.545358 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.545322 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-openshift-service-ca\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.546052 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.546026 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-openshift-service-ca\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.547686 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.547658 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-metrics-certs\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.549598 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.549577 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-maas-api-tls\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.567985 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.567958 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58h2n\" (UniqueName: \"kubernetes.io/projected/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-kube-api-access-58h2n\") pod \"maas-api-e2e-mt-67953efd-956bfdb9-z49lq\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:24.756126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:24.749624 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:25.113339 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:41:25.113307 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68a0ff5_8958_4860_bcba_71f0e4a89d7d.slice/crio-8adbc804cc7f65f1d725623ee6004739ec66e8060888c1b690b79c373ca4b558 WatchSource:0}: Error finding container 8adbc804cc7f65f1d725623ee6004739ec66e8060888c1b690b79c373ca4b558: Status 404 returned error can't find the container with id 8adbc804cc7f65f1d725623ee6004739ec66e8060888c1b690b79c373ca4b558 Oct 30 19:41:25.115320 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:25.115297 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq"] Oct 30 19:41:25.260024 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:25.259989 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" event={"ID":"b68a0ff5-8958-4860-bcba-71f0e4a89d7d","Type":"ContainerStarted","Data":"8adbc804cc7f65f1d725623ee6004739ec66e8060888c1b690b79c373ca4b558"} Oct 30 19:41:26.270727 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:26.270688 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" event={"ID":"b68a0ff5-8958-4860-bcba-71f0e4a89d7d","Type":"ContainerStarted","Data":"16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1"} Oct 30 19:41:26.271259 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:26.270835 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:26.320194 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:26.320133 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" podStartSLOduration=2.017621996 podStartE2EDuration="2.320095808s" totalImagesPullingTime="302.473812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:26.318767437 +0000 UTC m=+1247.232372055" watchObservedRunningTime="2026-10-30 19:41:26.320095808 +0000 UTC m=+1247.233700426" Oct 30 19:41:32.303918 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:32.303863 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:41:33.035243 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.035208 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz"] Oct 30 19:41:33.040212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.040189 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.062227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.062198 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz"] Oct 30 19:41:33.076206 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.076181 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-fe4ec3-serving-cert\"" Oct 30 19:41:33.140218 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.140183 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.140411 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.140254 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d727b1d-8115-4618-9d22-2c320c1c9800-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.140469 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.140432 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.140568 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.140547 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwpjn\" (UniqueName: \"kubernetes.io/projected/8d727b1d-8115-4618-9d22-2c320c1c9800-kube-api-access-rwpjn\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.241730 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.241698 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwpjn\" (UniqueName: \"kubernetes.io/projected/8d727b1d-8115-4618-9d22-2c320c1c9800-kube-api-access-rwpjn\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.241901 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.241806 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.241901 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.241841 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d727b1d-8115-4618-9d22-2c320c1c9800-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.241994 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:41:33.241962 4904 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-serving-cert: secret "maas-api-e2e-shared-a-w1-fe4ec3-serving-cert" not found Oct 30 19:41:33.242089 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:41:33.242043 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-maas-api-tls podName:8d727b1d-8115-4618-9d22-2c320c1c9800 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:33.742022099 +0000 UTC m=+1254.655626694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-maas-api-tls") pod "maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" (UID: "8d727b1d-8115-4618-9d22-2c320c1c9800") : secret "maas-api-e2e-shared-a-w1-fe4ec3-serving-cert" not found Oct 30 19:41:33.242089 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.242067 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.242489 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.242470 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d727b1d-8115-4618-9d22-2c320c1c9800-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.245184 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.245153 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.269126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.267918 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwpjn\" (UniqueName: \"kubernetes.io/projected/8d727b1d-8115-4618-9d22-2c320c1c9800-kube-api-access-rwpjn\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.759700 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.759649 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.763684 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.762995 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:33.972364 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:33.972329 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:34.619664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:34.619633 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz"] Oct 30 19:41:35.407751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:35.407672 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" event={"ID":"8d727b1d-8115-4618-9d22-2c320c1c9800","Type":"ContainerStarted","Data":"9b9bb7ff7d8a21b47f697988a668e3c12895623380509226cff2f1d521a09ee0"} Oct 30 19:41:35.407751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:35.407737 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:35.408181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:35.407762 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" event={"ID":"8d727b1d-8115-4618-9d22-2c320c1c9800","Type":"ContainerStarted","Data":"841d60993a1a6b499433a5c38dc19be35dab50110c5c1380688b0b31cc3baea5"} Oct 30 19:41:35.464551 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:35.464507 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" podStartSLOduration=2.047536911 podStartE2EDuration="2.46449425s" totalImagesPullingTime="416.957339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:35.45997338 +0000 UTC m=+1256.373577997" watchObservedRunningTime="2026-10-30 19:41:35.46449425 +0000 UTC m=+1256.378098867" Oct 30 19:41:39.628915 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.628888 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg"] Oct 30 19:41:39.633602 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.633582 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.656658 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.656631 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7e89da-serving-cert\"" Oct 30 19:41:39.667836 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.667812 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg"] Oct 30 19:41:39.747500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.747467 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbwh\" (UniqueName: \"kubernetes.io/projected/4da7f290-6c92-4372-9104-6b6530fcddb5-kube-api-access-phbwh\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.747631 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.747522 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.747631 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.747606 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4da7f290-6c92-4372-9104-6b6530fcddb5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.747757 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.747744 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.851883 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.851838 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbwh\" (UniqueName: \"kubernetes.io/projected/4da7f290-6c92-4372-9104-6b6530fcddb5-kube-api-access-phbwh\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.852084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.851943 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.852084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.852027 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4da7f290-6c92-4372-9104-6b6530fcddb5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.852240 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:41:39.852086 4904 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-serving-cert: secret "maas-api-e2e-shared-b-w3-7e89da-serving-cert" not found Oct 30 19:41:39.852240 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:41:39.852208 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-maas-api-tls podName:4da7f290-6c92-4372-9104-6b6530fcddb5 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:40.352184911 +0000 UTC m=+1261.265789526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" (UID: "4da7f290-6c92-4372-9104-6b6530fcddb5") : secret "maas-api-e2e-shared-b-w3-7e89da-serving-cert" not found Oct 30 19:41:39.852391 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.852303 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.853012 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.852972 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4da7f290-6c92-4372-9104-6b6530fcddb5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.864852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.864808 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:39.876481 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:39.876457 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbwh\" (UniqueName: \"kubernetes.io/projected/4da7f290-6c92-4372-9104-6b6530fcddb5-kube-api-access-phbwh\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:40.362621 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:40.362590 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:40.367239 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:40.367211 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:40.569997 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:40.569957 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:40.972213 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:40.971976 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg"] Oct 30 19:41:40.982942 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:41:40.982913 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da7f290_6c92_4372_9104_6b6530fcddb5.slice/crio-7af680fde6fa04ffe2697e97c867e1292f84d7a729490f924285ca16edbffc1b WatchSource:0}: Error finding container 7af680fde6fa04ffe2697e97c867e1292f84d7a729490f924285ca16edbffc1b: Status 404 returned error can't find the container with id 7af680fde6fa04ffe2697e97c867e1292f84d7a729490f924285ca16edbffc1b Oct 30 19:41:41.435775 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:41.434544 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:41:41.477010 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:41.476977 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" event={"ID":"4da7f290-6c92-4372-9104-6b6530fcddb5","Type":"ContainerStarted","Data":"7af680fde6fa04ffe2697e97c867e1292f84d7a729490f924285ca16edbffc1b"} Oct 30 19:41:42.486696 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:42.486665 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" event={"ID":"4da7f290-6c92-4372-9104-6b6530fcddb5","Type":"ContainerStarted","Data":"244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd"} Oct 30 19:41:42.487097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:42.486738 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:42.547156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:42.547078 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" podStartSLOduration=3.155717944 podStartE2EDuration="3.547061242s" totalImagesPullingTime="391.343298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:42.540025031 +0000 UTC m=+1263.453629649" watchObservedRunningTime="2026-10-30 19:41:42.547061242 +0000 UTC m=+1263.460665859" Oct 30 19:41:48.499404 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:48.499374 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:41:54.514017 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:41:54.513977 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:41:55.041833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:55.041802 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp"] Oct 30 19:41:55.053828 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:55.053802 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d3ed095-h6xdp"] Oct 30 19:41:55.224993 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:41:55.224964 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83190974-909e-4e8e-869e-1db259d1fe17" path="/var/lib/kubelet/pods/83190974-909e-4e8e-869e-1db259d1fe17/volumes" Oct 30 19:42:01.447489 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:01.447451 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc"] Oct 30 19:42:01.454177 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:01.451416 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" containerID="cri-o://bdc7c17e06dc273b7a883794a23921dda54651d179f39da797a1ff1f779b650f" gracePeriod=30 Oct 30 19:42:03.389508 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.389470 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v"] Oct 30 19:42:03.395540 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.395517 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.404331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.404307 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v"] Oct 30 19:42:03.431167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.431134 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-metrics-certs\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.431312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.431274 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-maas-api-tls\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.431379 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.431366 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d87143-d4c2-483a-adc6-e79c448a708b-openshift-service-ca\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.431457 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.431442 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm6vp\" (UniqueName: \"kubernetes.io/projected/22d87143-d4c2-483a-adc6-e79c448a708b-kube-api-access-rm6vp\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.460296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.460275 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4a4e2f-serving-cert\"" Oct 30 19:42:03.532669 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.532617 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-metrics-certs\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.532951 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.532931 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-maas-api-tls\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.533072 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.533052 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d87143-d4c2-483a-adc6-e79c448a708b-openshift-service-ca\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.533288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.533269 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm6vp\" (UniqueName: \"kubernetes.io/projected/22d87143-d4c2-483a-adc6-e79c448a708b-kube-api-access-rm6vp\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.534185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.534162 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d87143-d4c2-483a-adc6-e79c448a708b-openshift-service-ca\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.536848 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.536825 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-metrics-certs\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.537032 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.537008 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-maas-api-tls\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.560266 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.560242 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm6vp\" (UniqueName: \"kubernetes.io/projected/22d87143-d4c2-483a-adc6-e79c448a708b-kube-api-access-rm6vp\") pod \"maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:03.715333 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:03.715296 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:04.082582 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:42:04.082553 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d87143_d4c2_483a_adc6_e79c448a708b.slice/crio-8ff1bc16ff9b0ef01006e5d159bd904b93e039a51ebd56c7d6dbbdbfd1a91252 WatchSource:0}: Error finding container 8ff1bc16ff9b0ef01006e5d159bd904b93e039a51ebd56c7d6dbbdbfd1a91252: Status 404 returned error can't find the container with id 8ff1bc16ff9b0ef01006e5d159bd904b93e039a51ebd56c7d6dbbdbfd1a91252 Oct 30 19:42:04.083478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:04.083454 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v"] Oct 30 19:42:04.083966 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:04.083951 4904 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:42:04.725092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:04.725053 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" event={"ID":"22d87143-d4c2-483a-adc6-e79c448a708b","Type":"ContainerStarted","Data":"f3f5650df557b373ce8b91285a4d01e7872c32a14996666ae720ffe8345a3c51"} Oct 30 19:42:04.725809 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:04.725137 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" event={"ID":"22d87143-d4c2-483a-adc6-e79c448a708b","Type":"ContainerStarted","Data":"8ff1bc16ff9b0ef01006e5d159bd904b93e039a51ebd56c7d6dbbdbfd1a91252"} Oct 30 19:42:04.726326 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:04.726261 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:04.771250 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:04.771141 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" podStartSLOduration=1.376328693 podStartE2EDuration="1.771125222s" totalImagesPullingTime="394.796529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:04.770648512 +0000 UTC m=+1285.684253130" watchObservedRunningTime="2026-10-30 19:42:04.771125222 +0000 UTC m=+1285.684729840" Oct 30 19:42:06.760147 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.760066 4904 generic.go:356] "Generic (PLEG): container finished" podID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerID="bdc7c17e06dc273b7a883794a23921dda54651d179f39da797a1ff1f779b650f" exitCode=0 Oct 30 19:42:06.761313 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.761286 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" event={"ID":"01b09245-d0ba-4db8-b725-075c70cb3b90","Type":"ContainerDied","Data":"bdc7c17e06dc273b7a883794a23921dda54651d179f39da797a1ff1f779b650f"} Oct 30 19:42:06.791332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.791312 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:42:06.919131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.918212 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-kube-api-access-sf8d8\" (UniqueName: \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-kube-api-access-sf8d8\") pod \"01b09245-d0ba-4db8-b725-075c70cb3b90\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " Oct 30 19:42:06.919131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.918791 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-podinfo\") pod \"01b09245-d0ba-4db8-b725-075c70cb3b90\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.919181 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-envoy\") pod \"01b09245-d0ba-4db8-b725-075c70cb3b90\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.919418 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-token\" (UniqueName: \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-token\") pod \"01b09245-d0ba-4db8-b725-075c70cb3b90\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.919777 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-socket" pod "01b09245-d0ba-4db8-b725-075c70cb3b90" (UID: "01b09245-d0ba-4db8-b725-075c70cb3b90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.919463 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-socket\") pod \"01b09245-d0ba-4db8-b725-075c70cb3b90\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.920493 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-data" pod "01b09245-d0ba-4db8-b725-075c70cb3b90" (UID: "01b09245-d0ba-4db8-b725-075c70cb3b90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.920526 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-data\") pod \"01b09245-d0ba-4db8-b725-075c70cb3b90\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.920783 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b09245-d0ba-4db8-b725-075c70cb3b90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b09245-d0ba-4db8-b725-075c70cb3b90-istiod-ca-cert\") pod \"01b09245-d0ba-4db8-b725-075c70cb3b90\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.921166 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-certs" pod "01b09245-d0ba-4db8-b725-075c70cb3b90" (UID: "01b09245-d0ba-4db8-b725-075c70cb3b90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.921435 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b09245-d0ba-4db8-b725-075c70cb3b90-istiod-ca-cert" pod "01b09245-d0ba-4db8-b725-075c70cb3b90" (UID: "01b09245-d0ba-4db8-b725-075c70cb3b90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.920897 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-certs\") pod \"01b09245-d0ba-4db8-b725-075c70cb3b90\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.921568 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-credential-socket\") pod \"01b09245-d0ba-4db8-b725-075c70cb3b90\" (UID: \"01b09245-d0ba-4db8-b725-075c70cb3b90\") " Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.922351 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-credential-socket" pod "01b09245-d0ba-4db8-b725-075c70cb3b90" (UID: "01b09245-d0ba-4db8-b725-075c70cb3b90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.922590 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.922609 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.922621 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b09245-d0ba-4db8-b725-075c70cb3b90-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.922633 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.928299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.922645 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.935773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.935743 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-kube-api-access-sf8d8" pod "01b09245-d0ba-4db8-b725-075c70cb3b90" (UID: "01b09245-d0ba-4db8-b725-075c70cb3b90"). InnerVolumeSpecName "kube-api-access-sf8d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:06.938200 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.938173 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-podinfo" pod "01b09245-d0ba-4db8-b725-075c70cb3b90" (UID: "01b09245-d0ba-4db8-b725-075c70cb3b90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:06.938485 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.938463 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-token" pod "01b09245-d0ba-4db8-b725-075c70cb3b90" (UID: "01b09245-d0ba-4db8-b725-075c70cb3b90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:06.938613 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:06.938594 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-envoy" pod "01b09245-d0ba-4db8-b725-075c70cb3b90" (UID: "01b09245-d0ba-4db8-b725-075c70cb3b90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:07.026478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:07.026398 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf8d8\" (UniqueName: \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-kube-api-access-sf8d8\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:07.026478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:07.026433 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:07.026478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:07.026449 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:07.026478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:07.026461 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b09245-d0ba-4db8-b725-075c70cb3b90-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:07.772867 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:07.772825 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" event={"ID":"01b09245-d0ba-4db8-b725-075c70cb3b90","Type":"ContainerDied","Data":"87579b0316e9a0e8af2b17b1d4329122982a80d16c615a877ea629ebe9fc6b9e"} Oct 30 19:42:07.773375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:07.772848 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc" Oct 30 19:42:07.773375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:07.772917 4904 scope.go:122] "RemoveContainer" containerID="bdc7c17e06dc273b7a883794a23921dda54651d179f39da797a1ff1f779b650f" Oct 30 19:42:07.849209 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:07.849176 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc"] Oct 30 19:42:07.863352 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:07.863325 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-sb6hc"] Oct 30 19:42:09.265123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:09.264229 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" path="/var/lib/kubelet/pods/01b09245-d0ba-4db8-b725-075c70cb3b90/volumes" Oct 30 19:42:11.767265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:11.767231 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:42:13.022802 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.022764 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-3bd4a5"] Oct 30 19:42:13.023976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.023948 4904 cpu_manager.go:414] "containerMap: removing container" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" Oct 30 19:42:13.023976 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.023973 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" Oct 30 19:42:13.024244 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.024200 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b09245-d0ba-4db8-b725-075c70cb3b90" containerName="istio-proxy" Oct 30 19:42:13.030224 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.030201 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-3bd4a5" Oct 30 19:42:13.040940 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.040913 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-3bd4a5"] Oct 30 19:42:13.046472 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.046443 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r79h9\"" Oct 30 19:42:13.122345 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.122313 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h67w\" (UniqueName: \"kubernetes.io/projected/a336295e-1ce8-40ac-863e-63eafa780d55-kube-api-access-7h67w\") pod \"test-curl-7257-3bd4a5\" (UID: \"a336295e-1ce8-40ac-863e-63eafa780d55\") " pod="opendatahub/test-curl-7257-3bd4a5" Oct 30 19:42:13.223593 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.223563 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h67w\" (UniqueName: \"kubernetes.io/projected/a336295e-1ce8-40ac-863e-63eafa780d55-kube-api-access-7h67w\") pod \"test-curl-7257-3bd4a5\" (UID: \"a336295e-1ce8-40ac-863e-63eafa780d55\") " pod="opendatahub/test-curl-7257-3bd4a5" Oct 30 19:42:13.246418 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.246389 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h67w\" (UniqueName: \"kubernetes.io/projected/a336295e-1ce8-40ac-863e-63eafa780d55-kube-api-access-7h67w\") pod \"test-curl-7257-3bd4a5\" (UID: \"a336295e-1ce8-40ac-863e-63eafa780d55\") " pod="opendatahub/test-curl-7257-3bd4a5" Oct 30 19:42:13.352497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.352429 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-3bd4a5" Oct 30 19:42:13.658448 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.658421 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-3bd4a5"] Oct 30 19:42:13.659201 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:42:13.659172 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda336295e_1ce8_40ac_863e_63eafa780d55.slice/crio-09eeb73013bdb8efd18c816f1edf71d3ed63e8c45f863fab4d38b088f67ac66c WatchSource:0}: Error finding container 09eeb73013bdb8efd18c816f1edf71d3ed63e8c45f863fab4d38b088f67ac66c: Status 404 returned error can't find the container with id 09eeb73013bdb8efd18c816f1edf71d3ed63e8c45f863fab4d38b088f67ac66c Oct 30 19:42:13.905496 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.905419 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-3bd4a5" event={"ID":"a336295e-1ce8-40ac-863e-63eafa780d55","Type":"ContainerStarted","Data":"98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1"} Oct 30 19:42:13.905496 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.905465 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-3bd4a5" event={"ID":"a336295e-1ce8-40ac-863e-63eafa780d55","Type":"ContainerStarted","Data":"09eeb73013bdb8efd18c816f1edf71d3ed63e8c45f863fab4d38b088f67ac66c"} Oct 30 19:42:13.949149 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:13.949085 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7257-3bd4a5" podStartSLOduration=1.949071983 podStartE2EDuration="1.949071983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:13.945222107 +0000 UTC m=+1294.858826723" watchObservedRunningTime="2026-10-30 19:42:13.949071983 +0000 UTC m=+1294.862676601" Oct 30 19:42:14.777638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:14.774893 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-3bd4a5"] Oct 30 19:42:14.786594 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:14.786561 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-3bd4a5"] Oct 30 19:42:14.993408 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:14.993376 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-486ae5"] Oct 30 19:42:14.994180 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:14.994158 4904 cpu_manager.go:414] "containerMap: removing container" podUID="a336295e-1ce8-40ac-863e-63eafa780d55" containerName="test-curl-7257-3bd4a5" Oct 30 19:42:14.994180 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:14.994180 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a336295e-1ce8-40ac-863e-63eafa780d55" containerName="test-curl-7257-3bd4a5" Oct 30 19:42:14.994395 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:14.994382 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="a336295e-1ce8-40ac-863e-63eafa780d55" containerName="test-curl-7257-3bd4a5" Oct 30 19:42:14.998088 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:14.998070 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-486ae5" Oct 30 19:42:15.036860 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:15.036806 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-486ae5"] Oct 30 19:42:15.055351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:15.055324 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjwf\" (UniqueName: \"kubernetes.io/projected/9a5364e4-574e-4c9e-a549-3ea8d8c9368d-kube-api-access-qnjwf\") pod \"test-curl-7257-486ae5\" (UID: \"9a5364e4-574e-4c9e-a549-3ea8d8c9368d\") " pod="opendatahub/test-curl-7257-486ae5" Oct 30 19:42:15.156812 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:15.156773 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjwf\" (UniqueName: \"kubernetes.io/projected/9a5364e4-574e-4c9e-a549-3ea8d8c9368d-kube-api-access-qnjwf\") pod \"test-curl-7257-486ae5\" (UID: \"9a5364e4-574e-4c9e-a549-3ea8d8c9368d\") " pod="opendatahub/test-curl-7257-486ae5" Oct 30 19:42:15.199870 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:15.199837 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjwf\" (UniqueName: \"kubernetes.io/projected/9a5364e4-574e-4c9e-a549-3ea8d8c9368d-kube-api-access-qnjwf\") pod \"test-curl-7257-486ae5\" (UID: \"9a5364e4-574e-4c9e-a549-3ea8d8c9368d\") " pod="opendatahub/test-curl-7257-486ae5" Oct 30 19:42:15.311346 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:15.311273 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-486ae5" Oct 30 19:42:15.680303 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:15.680276 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-486ae5"] Oct 30 19:42:15.680458 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:42:15.680439 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a5364e4_574e_4c9e_a549_3ea8d8c9368d.slice/crio-f9784fbd2697c8a190a81a5a3c504ac1319614a7980ce43fca613aa5e8d87a1e WatchSource:0}: Error finding container f9784fbd2697c8a190a81a5a3c504ac1319614a7980ce43fca613aa5e8d87a1e: Status 404 returned error can't find the container with id f9784fbd2697c8a190a81a5a3c504ac1319614a7980ce43fca613aa5e8d87a1e Oct 30 19:42:15.934301 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:15.934187 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-486ae5" event={"ID":"9a5364e4-574e-4c9e-a549-3ea8d8c9368d","Type":"ContainerStarted","Data":"4000c82d0abfc5ebe0e6d03a015f9a02a548527edd9941e801204feddc9ef29d"} Oct 30 19:42:15.934301 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:15.934282 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-486ae5" event={"ID":"9a5364e4-574e-4c9e-a549-3ea8d8c9368d","Type":"ContainerStarted","Data":"f9784fbd2697c8a190a81a5a3c504ac1319614a7980ce43fca613aa5e8d87a1e"} Oct 30 19:42:15.934728 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:15.934525 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7257-3bd4a5" podUID="a336295e-1ce8-40ac-863e-63eafa780d55" containerName="test-curl-7257-3bd4a5" containerID="cri-o://98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1" gracePeriod=2 Oct 30 19:42:15.976879 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:15.976827 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7257-486ae5" podStartSLOduration=1.97681191 podStartE2EDuration="1.97681191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:15.975623256 +0000 UTC m=+1296.889227873" watchObservedRunningTime="2026-10-30 19:42:15.97681191 +0000 UTC m=+1296.890416532" Oct 30 19:42:16.932654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:16.932617 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-486ae5"] Oct 30 19:42:16.940987 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:16.939432 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-486ae5"] Oct 30 19:42:17.108289 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.108257 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-6b6d54"] Oct 30 19:42:17.109027 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.109006 4904 cpu_manager.go:414] "containerMap: removing container" podUID="9a5364e4-574e-4c9e-a549-3ea8d8c9368d" containerName="test-curl-7257-486ae5" Oct 30 19:42:17.109027 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.109026 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5364e4-574e-4c9e-a549-3ea8d8c9368d" containerName="test-curl-7257-486ae5" Oct 30 19:42:17.109178 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.109155 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5364e4-574e-4c9e-a549-3ea8d8c9368d" containerName="test-curl-7257-486ae5" Oct 30 19:42:17.113068 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.113049 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-6b6d54" Oct 30 19:42:17.126967 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.126941 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-6b6d54"] Oct 30 19:42:17.189731 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.189670 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmxds\" (UniqueName: \"kubernetes.io/projected/817be56a-56de-4550-a01a-111d8cc6e4d4-kube-api-access-vmxds\") pod \"test-curl-7257-6b6d54\" (UID: \"817be56a-56de-4550-a01a-111d8cc6e4d4\") " pod="opendatahub/test-curl-7257-6b6d54" Oct 30 19:42:17.292824 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.292787 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmxds\" (UniqueName: \"kubernetes.io/projected/817be56a-56de-4550-a01a-111d8cc6e4d4-kube-api-access-vmxds\") pod \"test-curl-7257-6b6d54\" (UID: \"817be56a-56de-4550-a01a-111d8cc6e4d4\") " pod="opendatahub/test-curl-7257-6b6d54" Oct 30 19:42:17.318478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.318451 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmxds\" (UniqueName: \"kubernetes.io/projected/817be56a-56de-4550-a01a-111d8cc6e4d4-kube-api-access-vmxds\") pod \"test-curl-7257-6b6d54\" (UID: \"817be56a-56de-4550-a01a-111d8cc6e4d4\") " pod="opendatahub/test-curl-7257-6b6d54" Oct 30 19:42:17.427053 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.427025 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-6b6d54" Oct 30 19:42:17.741843 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:42:17.741810 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod817be56a_56de_4550_a01a_111d8cc6e4d4.slice/crio-aace91c686551211b98f1196bdddbbda672440384a9a8aefd3a580509bcc1127 WatchSource:0}: Error finding container aace91c686551211b98f1196bdddbbda672440384a9a8aefd3a580509bcc1127: Status 404 returned error can't find the container with id aace91c686551211b98f1196bdddbbda672440384a9a8aefd3a580509bcc1127 Oct 30 19:42:17.742753 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.742735 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-6b6d54"] Oct 30 19:42:17.958174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.958146 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-6b6d54" event={"ID":"817be56a-56de-4550-a01a-111d8cc6e4d4","Type":"ContainerStarted","Data":"48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678"} Oct 30 19:42:17.958544 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.958186 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-6b6d54" event={"ID":"817be56a-56de-4550-a01a-111d8cc6e4d4","Type":"ContainerStarted","Data":"aace91c686551211b98f1196bdddbbda672440384a9a8aefd3a580509bcc1127"} Oct 30 19:42:17.958544 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:17.958273 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7257-486ae5" podUID="9a5364e4-574e-4c9e-a549-3ea8d8c9368d" containerName="test-curl-7257-486ae5" containerID="cri-o://4000c82d0abfc5ebe0e6d03a015f9a02a548527edd9941e801204feddc9ef29d" gracePeriod=2 Oct 30 19:42:18.011397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.011309 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7257-6b6d54" podStartSLOduration=1.011297733 podStartE2EDuration="1.011297733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:18.009624121 +0000 UTC m=+1298.923228738" watchObservedRunningTime="2026-10-30 19:42:18.011297733 +0000 UTC m=+1298.924902349" Oct 30 19:42:18.142820 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.142796 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-3bd4a5" Oct 30 19:42:18.310007 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.309912 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a336295e-1ce8-40ac-863e-63eafa780d55-kube-api-access-7h67w\" (UniqueName: \"kubernetes.io/projected/a336295e-1ce8-40ac-863e-63eafa780d55-kube-api-access-7h67w\") pod \"a336295e-1ce8-40ac-863e-63eafa780d55\" (UID: \"a336295e-1ce8-40ac-863e-63eafa780d55\") " Oct 30 19:42:18.312781 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.312749 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a336295e-1ce8-40ac-863e-63eafa780d55-kube-api-access-7h67w" pod "a336295e-1ce8-40ac-863e-63eafa780d55" (UID: "a336295e-1ce8-40ac-863e-63eafa780d55"). InnerVolumeSpecName "kube-api-access-7h67w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:18.414735 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.412785 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h67w\" (UniqueName: \"kubernetes.io/projected/a336295e-1ce8-40ac-863e-63eafa780d55-kube-api-access-7h67w\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:18.954227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.954190 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-6b6d54"] Oct 30 19:42:18.959687 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.959576 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-6b6d54"] Oct 30 19:42:18.978141 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.978062 4904 generic.go:356] "Generic (PLEG): container finished" podID="a336295e-1ce8-40ac-863e-63eafa780d55" containerID="98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1" exitCode=137 Oct 30 19:42:18.979261 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.979237 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-3bd4a5" Oct 30 19:42:18.981201 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.981181 4904 scope.go:122] "RemoveContainer" containerID="98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1" Oct 30 19:42:18.997853 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.997835 4904 scope.go:122] "RemoveContainer" containerID="98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1" Oct 30 19:42:18.998161 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:18.998142 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1\": container with ID starting with 98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1 not found: ID does not exist" containerID="98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1" Oct 30 19:42:18.998252 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:18.998173 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1"} err="failed to get container status \"98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1\": rpc error: code = NotFound desc = could not find container \"98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1\": container with ID starting with 98e21c603f87c79bb81ed1712c252caa1d3bcef1e2856529af76a0a53e9159b1 not found: ID does not exist" Oct 30 19:42:19.008613 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:19.008581 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-3bd4a5\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="a336295e-1ce8-40ac-863e-63eafa780d55" pod="opendatahub/test-curl-7257-3bd4a5" Oct 30 19:42:19.115675 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.115648 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-be6ab7"] Oct 30 19:42:19.116642 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.116624 4904 cpu_manager.go:414] "containerMap: removing container" podUID="817be56a-56de-4550-a01a-111d8cc6e4d4" containerName="test-curl-7257-6b6d54" Oct 30 19:42:19.116691 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.116645 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="817be56a-56de-4550-a01a-111d8cc6e4d4" containerName="test-curl-7257-6b6d54" Oct 30 19:42:19.116849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.116836 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="817be56a-56de-4550-a01a-111d8cc6e4d4" containerName="test-curl-7257-6b6d54" Oct 30 19:42:19.121826 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.121808 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-be6ab7" Oct 30 19:42:19.133709 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.133688 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-be6ab7"] Oct 30 19:42:19.178077 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:19.178045 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-3bd4a5\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="a336295e-1ce8-40ac-863e-63eafa780d55" pod="opendatahub/test-curl-7257-3bd4a5" Oct 30 19:42:19.226061 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.226004 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a336295e-1ce8-40ac-863e-63eafa780d55" path="/var/lib/kubelet/pods/a336295e-1ce8-40ac-863e-63eafa780d55/volumes" Oct 30 19:42:19.226429 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.226396 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7b6d\" (UniqueName: \"kubernetes.io/projected/05db9e55-6a64-4e49-a697-555333da80cf-kube-api-access-t7b6d\") pod \"test-curl-7257-be6ab7\" (UID: \"05db9e55-6a64-4e49-a697-555333da80cf\") " pod="opendatahub/test-curl-7257-be6ab7" Oct 30 19:42:19.236308 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:19.236282 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-3bd4a5\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="a336295e-1ce8-40ac-863e-63eafa780d55" pod="opendatahub/test-curl-7257-3bd4a5" Oct 30 19:42:19.327309 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.327286 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7b6d\" (UniqueName: \"kubernetes.io/projected/05db9e55-6a64-4e49-a697-555333da80cf-kube-api-access-t7b6d\") pod \"test-curl-7257-be6ab7\" (UID: \"05db9e55-6a64-4e49-a697-555333da80cf\") " pod="opendatahub/test-curl-7257-be6ab7" Oct 30 19:42:19.348900 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.348874 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7b6d\" (UniqueName: \"kubernetes.io/projected/05db9e55-6a64-4e49-a697-555333da80cf-kube-api-access-t7b6d\") pod \"test-curl-7257-be6ab7\" (UID: \"05db9e55-6a64-4e49-a697-555333da80cf\") " pod="opendatahub/test-curl-7257-be6ab7" Oct 30 19:42:19.436052 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.436025 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-be6ab7" Oct 30 19:42:19.747930 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:42:19.747852 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05db9e55_6a64_4e49_a697_555333da80cf.slice/crio-15b75b70d1ce4d036f61813a529ea8cfdf5f9187633105cc2184ecbb16e9fa1f WatchSource:0}: Error finding container 15b75b70d1ce4d036f61813a529ea8cfdf5f9187633105cc2184ecbb16e9fa1f: Status 404 returned error can't find the container with id 15b75b70d1ce4d036f61813a529ea8cfdf5f9187633105cc2184ecbb16e9fa1f Oct 30 19:42:19.751071 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.750951 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-be6ab7"] Oct 30 19:42:19.997118 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.997071 4904 generic.go:356] "Generic (PLEG): container finished" podID="9a5364e4-574e-4c9e-a549-3ea8d8c9368d" containerID="4000c82d0abfc5ebe0e6d03a015f9a02a548527edd9941e801204feddc9ef29d" exitCode=137 Oct 30 19:42:19.999174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.999067 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-be6ab7" event={"ID":"05db9e55-6a64-4e49-a697-555333da80cf","Type":"ContainerStarted","Data":"61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff"} Oct 30 19:42:19.999174 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.999148 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-be6ab7" event={"ID":"05db9e55-6a64-4e49-a697-555333da80cf","Type":"ContainerStarted","Data":"15b75b70d1ce4d036f61813a529ea8cfdf5f9187633105cc2184ecbb16e9fa1f"} Oct 30 19:42:19.999344 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:19.999320 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7257-6b6d54" podUID="817be56a-56de-4550-a01a-111d8cc6e4d4" containerName="test-curl-7257-6b6d54" containerID="cri-o://48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678" gracePeriod=2 Oct 30 19:42:20.044743 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:20.044305 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7257-be6ab7" podStartSLOduration=1.044288146 podStartE2EDuration="1.044288146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:20.040653458 +0000 UTC m=+1300.954258073" watchObservedRunningTime="2026-10-30 19:42:20.044288146 +0000 UTC m=+1300.957892762" Oct 30 19:42:20.169356 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:20.169332 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-486ae5" Oct 30 19:42:20.343459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:20.343379 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5364e4-574e-4c9e-a549-3ea8d8c9368d-kube-api-access-qnjwf\" (UniqueName: \"kubernetes.io/projected/9a5364e4-574e-4c9e-a549-3ea8d8c9368d-kube-api-access-qnjwf\") pod \"9a5364e4-574e-4c9e-a549-3ea8d8c9368d\" (UID: \"9a5364e4-574e-4c9e-a549-3ea8d8c9368d\") " Oct 30 19:42:20.345614 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:20.345589 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5364e4-574e-4c9e-a549-3ea8d8c9368d-kube-api-access-qnjwf" pod "9a5364e4-574e-4c9e-a549-3ea8d8c9368d" (UID: "9a5364e4-574e-4c9e-a549-3ea8d8c9368d"). InnerVolumeSpecName "kube-api-access-qnjwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:20.445432 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:20.445403 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnjwf\" (UniqueName: \"kubernetes.io/projected/9a5364e4-574e-4c9e-a549-3ea8d8c9368d-kube-api-access-qnjwf\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:20.916983 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:20.916949 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-be6ab7"] Oct 30 19:42:20.924657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:20.924616 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-be6ab7"] Oct 30 19:42:21.036566 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:21.036539 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-486ae5" Oct 30 19:42:21.037411 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:21.037388 4904 scope.go:122] "RemoveContainer" containerID="4000c82d0abfc5ebe0e6d03a015f9a02a548527edd9941e801204feddc9ef29d" Oct 30 19:42:21.061551 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:21.061521 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-486ae5\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="9a5364e4-574e-4c9e-a549-3ea8d8c9368d" pod="opendatahub/test-curl-7257-486ae5" Oct 30 19:42:21.230444 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:21.230401 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5364e4-574e-4c9e-a549-3ea8d8c9368d" path="/var/lib/kubelet/pods/9a5364e4-574e-4c9e-a549-3ea8d8c9368d/volumes" Oct 30 19:42:22.047541 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:22.047455 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7257-be6ab7" podUID="05db9e55-6a64-4e49-a697-555333da80cf" containerName="test-curl-7257-be6ab7" containerID="cri-o://61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff" gracePeriod=2 Oct 30 19:42:22.199734 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:22.199709 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-6b6d54" Oct 30 19:42:22.275903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:22.275826 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/817be56a-56de-4550-a01a-111d8cc6e4d4-kube-api-access-vmxds\" (UniqueName: \"kubernetes.io/projected/817be56a-56de-4550-a01a-111d8cc6e4d4-kube-api-access-vmxds\") pod \"817be56a-56de-4550-a01a-111d8cc6e4d4\" (UID: \"817be56a-56de-4550-a01a-111d8cc6e4d4\") " Oct 30 19:42:22.278170 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:22.278142 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817be56a-56de-4550-a01a-111d8cc6e4d4-kube-api-access-vmxds" pod "817be56a-56de-4550-a01a-111d8cc6e4d4" (UID: "817be56a-56de-4550-a01a-111d8cc6e4d4"). InnerVolumeSpecName "kube-api-access-vmxds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:22.378671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:22.378640 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmxds\" (UniqueName: \"kubernetes.io/projected/817be56a-56de-4550-a01a-111d8cc6e4d4-kube-api-access-vmxds\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:23.058956 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:23.058920 4904 generic.go:356] "Generic (PLEG): container finished" podID="817be56a-56de-4550-a01a-111d8cc6e4d4" containerID="48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678" exitCode=137 Oct 30 19:42:23.059455 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:23.058979 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-6b6d54" Oct 30 19:42:23.059455 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:23.059023 4904 scope.go:122] "RemoveContainer" containerID="48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678" Oct 30 19:42:23.072901 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:23.072887 4904 scope.go:122] "RemoveContainer" containerID="48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678" Oct 30 19:42:23.073334 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:23.073315 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678\": container with ID starting with 48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678 not found: ID does not exist" containerID="48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678" Oct 30 19:42:23.073488 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:23.073465 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678"} err="failed to get container status \"48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678\": rpc error: code = NotFound desc = could not find container \"48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678\": container with ID starting with 48ad47640431f23e0186a228fe9611f34113445eff43af5d4dd53128c483b678 not found: ID does not exist" Oct 30 19:42:23.083664 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:23.083615 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-6b6d54\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="817be56a-56de-4550-a01a-111d8cc6e4d4" pod="opendatahub/test-curl-7257-6b6d54" Oct 30 19:42:23.230529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:23.230490 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="817be56a-56de-4550-a01a-111d8cc6e4d4" path="/var/lib/kubelet/pods/817be56a-56de-4550-a01a-111d8cc6e4d4/volumes" Oct 30 19:42:24.235183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:24.235159 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-be6ab7" Oct 30 19:42:24.311639 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:24.311559 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05db9e55-6a64-4e49-a697-555333da80cf-kube-api-access-t7b6d\" (UniqueName: \"kubernetes.io/projected/05db9e55-6a64-4e49-a697-555333da80cf-kube-api-access-t7b6d\") pod \"05db9e55-6a64-4e49-a697-555333da80cf\" (UID: \"05db9e55-6a64-4e49-a697-555333da80cf\") " Oct 30 19:42:24.313767 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:24.313741 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05db9e55-6a64-4e49-a697-555333da80cf-kube-api-access-t7b6d" pod "05db9e55-6a64-4e49-a697-555333da80cf" (UID: "05db9e55-6a64-4e49-a697-555333da80cf"). InnerVolumeSpecName "kube-api-access-t7b6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:24.413655 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:24.413622 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7b6d\" (UniqueName: \"kubernetes.io/projected/05db9e55-6a64-4e49-a697-555333da80cf-kube-api-access-t7b6d\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:42:24.511934 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:24.511897 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:42:25.078757 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:25.078724 4904 generic.go:356] "Generic (PLEG): container finished" podID="05db9e55-6a64-4e49-a697-555333da80cf" containerID="61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff" exitCode=137 Oct 30 19:42:25.078940 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:25.078774 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-be6ab7" Oct 30 19:42:25.078940 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:25.078820 4904 scope.go:122] "RemoveContainer" containerID="61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff" Oct 30 19:42:25.091218 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:25.090985 4904 scope.go:122] "RemoveContainer" containerID="61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff" Oct 30 19:42:25.091317 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:25.091294 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff\": container with ID starting with 61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff not found: ID does not exist" containerID="61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff" Oct 30 19:42:25.091369 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:25.091331 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff"} err="failed to get container status \"61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff\": rpc error: code = NotFound desc = could not find container \"61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff\": container with ID starting with 61c1b781f246b0d0803867cbbe898f6bd00ca24a8d612ba4262c521b6ad10fff not found: ID does not exist" Oct 30 19:42:25.102434 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:25.102404 4904 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-be6ab7\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="05db9e55-6a64-4e49-a697-555333da80cf" pod="opendatahub/test-curl-7257-be6ab7" Oct 30 19:42:25.224264 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:25.224231 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05db9e55-6a64-4e49-a697-555333da80cf" path="/var/lib/kubelet/pods/05db9e55-6a64-4e49-a697-555333da80cf/volumes" Oct 30 19:42:37.571126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.560279 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj"] Oct 30 19:42:37.571126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.561007 4904 cpu_manager.go:414] "containerMap: removing container" podUID="05db9e55-6a64-4e49-a697-555333da80cf" containerName="test-curl-7257-be6ab7" Oct 30 19:42:37.571126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.561021 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05db9e55-6a64-4e49-a697-555333da80cf" containerName="test-curl-7257-be6ab7" Oct 30 19:42:37.571126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.561174 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="05db9e55-6a64-4e49-a697-555333da80cf" containerName="test-curl-7257-be6ab7" Oct 30 19:42:37.571126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.565465 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.593926 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.593891 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-291d35-serving-cert\"" Oct 30 19:42:37.599490 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.599463 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj"] Oct 30 19:42:37.682683 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.682651 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.682868 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.682708 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.682868 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.682854 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eec7931-fe25-49a1-a11a-95040bbf7fba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.682986 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.682975 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zfw\" (UniqueName: \"kubernetes.io/projected/5eec7931-fe25-49a1-a11a-95040bbf7fba-kube-api-access-26zfw\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.784673 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.784640 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zfw\" (UniqueName: \"kubernetes.io/projected/5eec7931-fe25-49a1-a11a-95040bbf7fba-kube-api-access-26zfw\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.784867 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.784782 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.784867 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.784812 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.784985 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.784967 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eec7931-fe25-49a1-a11a-95040bbf7fba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.785078 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:37.785058 4904 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-serving-cert: secret "maas-api-e2e-shared-b-w1-291d35-serving-cert" not found Oct 30 19:42:37.785198 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:37.785184 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-maas-api-tls podName:5eec7931-fe25-49a1-a11a-95040bbf7fba nodeName:}" failed. No retries permitted until 2026-10-30 19:42:38.285162151 +0000 UTC m=+1319.198766750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-maas-api-tls") pod "maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" (UID: "5eec7931-fe25-49a1-a11a-95040bbf7fba") : secret "maas-api-e2e-shared-b-w1-291d35-serving-cert" not found Oct 30 19:42:37.785884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.785859 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eec7931-fe25-49a1-a11a-95040bbf7fba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.787518 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.787498 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:37.810751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:37.810714 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zfw\" (UniqueName: \"kubernetes.io/projected/5eec7931-fe25-49a1-a11a-95040bbf7fba-kube-api-access-26zfw\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:38.294237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:38.294200 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:38.296635 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:38.296604 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:38.490301 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:38.490269 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:38.920293 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:42:38.899513 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eec7931_fe25_49a1_a11a_95040bbf7fba.slice/crio-1c8f3488e06a106d6eab06d292cc561cad770dd04e3f2c17682079042271edb6 WatchSource:0}: Error finding container 1c8f3488e06a106d6eab06d292cc561cad770dd04e3f2c17682079042271edb6: Status 404 returned error can't find the container with id 1c8f3488e06a106d6eab06d292cc561cad770dd04e3f2c17682079042271edb6 Oct 30 19:42:38.920293 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:38.900736 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj"] Oct 30 19:42:39.327124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:39.312223 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" event={"ID":"5eec7931-fe25-49a1-a11a-95040bbf7fba","Type":"ContainerStarted","Data":"1c8f3488e06a106d6eab06d292cc561cad770dd04e3f2c17682079042271edb6"} Oct 30 19:42:40.300358 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:40.300322 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" event={"ID":"5eec7931-fe25-49a1-a11a-95040bbf7fba","Type":"ContainerStarted","Data":"b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2"} Oct 30 19:42:40.300774 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:40.300426 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:40.353444 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:40.353387 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" podStartSLOduration=2.786431222 podStartE2EDuration="3.353368314s" totalImagesPullingTime="566.937092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:40.346316253 +0000 UTC m=+1321.259920872" watchObservedRunningTime="2026-10-30 19:42:40.353368314 +0000 UTC m=+1321.266972932" Oct 30 19:42:46.313389 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:42:46.313360 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:42:54.555660 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:42:54.555591 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:43:01.991495 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:01.991455 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8"] Oct 30 19:43:01.991933 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:01.991662 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerName="istio-proxy" containerID="cri-o://1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada" gracePeriod=30 Oct 30 19:43:02.002688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:02.002656 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q"] Oct 30 19:43:02.002918 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:02.002894 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" containerID="cri-o://53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5" gracePeriod=30 Oct 30 19:43:02.010207 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:02.010176 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt"] Oct 30 19:43:02.012155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:02.010680 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" containerID="cri-o://849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf" gracePeriod=30 Oct 30 19:43:05.236995 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:05.236957 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:43:05.237528 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:05.237016 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:43:05.279334 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:05.279298 4904 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:43:05.279617 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:05.279587 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:43:07.428459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.428437 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:43:07.435836 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.435573 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:43:07.505888 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.505851 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:43:07.514524 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514491 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-certs\") pod \"47126122-ef5b-4e3f-9449-2fee7766f55a\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " Oct 30 19:43:07.514680 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514560 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-data\") pod \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " Oct 30 19:43:07.514680 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514610 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-data\") pod \"47126122-ef5b-4e3f-9449-2fee7766f55a\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " Oct 30 19:43:07.514680 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514637 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-credential-socket\") pod \"47126122-ef5b-4e3f-9449-2fee7766f55a\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " Oct 30 19:43:07.514680 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514675 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47126122-ef5b-4e3f-9449-2fee7766f55a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47126122-ef5b-4e3f-9449-2fee7766f55a-istiod-ca-cert\") pod \"47126122-ef5b-4e3f-9449-2fee7766f55a\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " Oct 30 19:43:07.514916 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514724 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-token\" (UniqueName: \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-token\") pod \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " Oct 30 19:43:07.514916 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514781 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-socket\") pod \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " Oct 30 19:43:07.514916 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514824 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-podinfo\") pod \"47126122-ef5b-4e3f-9449-2fee7766f55a\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " Oct 30 19:43:07.514916 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514868 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-envoy\") pod \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " Oct 30 19:43:07.515162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514934 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-envoy\") pod \"47126122-ef5b-4e3f-9449-2fee7766f55a\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " Oct 30 19:43:07.515162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.514970 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-token\" (UniqueName: \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-token\") pod \"47126122-ef5b-4e3f-9449-2fee7766f55a\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " Oct 30 19:43:07.515162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.515065 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istiod-ca-cert\") pod \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " Oct 30 19:43:07.515162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.515123 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-podinfo\") pod \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " Oct 30 19:43:07.515162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.515158 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-certs\") pod \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " Oct 30 19:43:07.515437 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.515196 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-kube-api-access-vvk4w\" (UniqueName: \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-kube-api-access-vvk4w\") pod \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " Oct 30 19:43:07.515437 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.515274 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-kube-api-access-gjhrh\" (UniqueName: \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-kube-api-access-gjhrh\") pod \"47126122-ef5b-4e3f-9449-2fee7766f55a\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " Oct 30 19:43:07.515437 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.515331 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-socket\") pod \"47126122-ef5b-4e3f-9449-2fee7766f55a\" (UID: \"47126122-ef5b-4e3f-9449-2fee7766f55a\") " Oct 30 19:43:07.515597 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.515449 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-credential-socket\") pod \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\" (UID: \"014713a3-4a5c-49ff-8417-be78ee3a1cc2\") " Oct 30 19:43:07.517570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.516257 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-credential-socket" pod "47126122-ef5b-4e3f-9449-2fee7766f55a" (UID: "47126122-ef5b-4e3f-9449-2fee7766f55a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.517570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.516488 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-certs" pod "47126122-ef5b-4e3f-9449-2fee7766f55a" (UID: "47126122-ef5b-4e3f-9449-2fee7766f55a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.517570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.516746 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.517570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.516766 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.517570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.516979 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-certs" pod "014713a3-4a5c-49ff-8417-be78ee3a1cc2" (UID: "014713a3-4a5c-49ff-8417-be78ee3a1cc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.517570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.517003 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-data" pod "014713a3-4a5c-49ff-8417-be78ee3a1cc2" (UID: "014713a3-4a5c-49ff-8417-be78ee3a1cc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.517570 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.517243 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-data" pod "47126122-ef5b-4e3f-9449-2fee7766f55a" (UID: "47126122-ef5b-4e3f-9449-2fee7766f55a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.519579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.518552 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istiod-ca-cert" pod "014713a3-4a5c-49ff-8417-be78ee3a1cc2" (UID: "014713a3-4a5c-49ff-8417-be78ee3a1cc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:07.519579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.518547 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-socket" pod "47126122-ef5b-4e3f-9449-2fee7766f55a" (UID: "47126122-ef5b-4e3f-9449-2fee7766f55a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.519579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.518820 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-credential-socket" pod "014713a3-4a5c-49ff-8417-be78ee3a1cc2" (UID: "014713a3-4a5c-49ff-8417-be78ee3a1cc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.519579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.519116 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47126122-ef5b-4e3f-9449-2fee7766f55a-istiod-ca-cert" pod "47126122-ef5b-4e3f-9449-2fee7766f55a" (UID: "47126122-ef5b-4e3f-9449-2fee7766f55a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:07.519579 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.519515 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-socket" pod "014713a3-4a5c-49ff-8417-be78ee3a1cc2" (UID: "014713a3-4a5c-49ff-8417-be78ee3a1cc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.522675 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.522650 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-token" pod "014713a3-4a5c-49ff-8417-be78ee3a1cc2" (UID: "014713a3-4a5c-49ff-8417-be78ee3a1cc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:07.525062 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.525026 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-podinfo" pod "014713a3-4a5c-49ff-8417-be78ee3a1cc2" (UID: "014713a3-4a5c-49ff-8417-be78ee3a1cc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:43:07.527167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.526973 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-podinfo" pod "47126122-ef5b-4e3f-9449-2fee7766f55a" (UID: "47126122-ef5b-4e3f-9449-2fee7766f55a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:43:07.527167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.527007 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-envoy" pod "014713a3-4a5c-49ff-8417-be78ee3a1cc2" (UID: "014713a3-4a5c-49ff-8417-be78ee3a1cc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.527167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.527079 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-token" pod "47126122-ef5b-4e3f-9449-2fee7766f55a" (UID: "47126122-ef5b-4e3f-9449-2fee7766f55a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:07.527167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.527094 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-envoy" pod "47126122-ef5b-4e3f-9449-2fee7766f55a" (UID: "47126122-ef5b-4e3f-9449-2fee7766f55a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.527610 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.527581 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-kube-api-access-vvk4w" pod "014713a3-4a5c-49ff-8417-be78ee3a1cc2" (UID: "014713a3-4a5c-49ff-8417-be78ee3a1cc2"). InnerVolumeSpecName "kube-api-access-vvk4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:07.528622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.528598 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-kube-api-access-gjhrh" pod "47126122-ef5b-4e3f-9449-2fee7766f55a" (UID: "47126122-ef5b-4e3f-9449-2fee7766f55a"). InnerVolumeSpecName "kube-api-access-gjhrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:07.617997 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.617907 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-podinfo\") pod \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " Oct 30 19:43:07.618187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.618016 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-data\") pod \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " Oct 30 19:43:07.618187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.618091 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istiod-ca-cert\") pod \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " Oct 30 19:43:07.618187 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.618169 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-envoy\") pod \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " Oct 30 19:43:07.618380 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.618264 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-socket\") pod \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " Oct 30 19:43:07.618380 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.618330 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-kube-api-access-8lqz7\" (UniqueName: \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-kube-api-access-8lqz7\") pod \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " Oct 30 19:43:07.618495 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.618454 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-certs\") pod \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " Oct 30 19:43:07.618557 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.618525 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-token\" (UniqueName: \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-token\") pod \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " Oct 30 19:43:07.618613 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.618560 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-credential-socket\") pod \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\" (UID: \"517ef3f2-04c5-4a53-8721-c5e44a9f494f\") " Oct 30 19:43:07.619393 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619363 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-certs" pod "517ef3f2-04c5-4a53-8721-c5e44a9f494f" (UID: "517ef3f2-04c5-4a53-8721-c5e44a9f494f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.619516 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619437 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619516 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619460 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619516 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619474 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619516 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619487 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619516 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619501 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619516 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619515 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvk4w\" (UniqueName: \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-kube-api-access-vvk4w\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619529 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjhrh\" (UniqueName: \"kubernetes.io/projected/47126122-ef5b-4e3f-9449-2fee7766f55a-kube-api-access-gjhrh\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619542 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619555 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619568 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619581 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619594 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47126122-ef5b-4e3f-9449-2fee7766f55a-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619606 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619619 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619630 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47126122-ef5b-4e3f-9449-2fee7766f55a-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619644 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014713a3-4a5c-49ff-8417-be78ee3a1cc2-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.619837 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619627 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-socket" pod "517ef3f2-04c5-4a53-8721-c5e44a9f494f" (UID: "517ef3f2-04c5-4a53-8721-c5e44a9f494f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.620425 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619906 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istiod-ca-cert" pod "517ef3f2-04c5-4a53-8721-c5e44a9f494f" (UID: "517ef3f2-04c5-4a53-8721-c5e44a9f494f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:07.620425 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.619962 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-credential-socket" pod "517ef3f2-04c5-4a53-8721-c5e44a9f494f" (UID: "517ef3f2-04c5-4a53-8721-c5e44a9f494f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.620425 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.620013 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-data" pod "517ef3f2-04c5-4a53-8721-c5e44a9f494f" (UID: "517ef3f2-04c5-4a53-8721-c5e44a9f494f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.622506 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.622477 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-kube-api-access-8lqz7" pod "517ef3f2-04c5-4a53-8721-c5e44a9f494f" (UID: "517ef3f2-04c5-4a53-8721-c5e44a9f494f"). InnerVolumeSpecName "kube-api-access-8lqz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:07.628371 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.628271 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-token" pod "517ef3f2-04c5-4a53-8721-c5e44a9f494f" (UID: "517ef3f2-04c5-4a53-8721-c5e44a9f494f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:07.628493 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.628389 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-envoy" pod "517ef3f2-04c5-4a53-8721-c5e44a9f494f" (UID: "517ef3f2-04c5-4a53-8721-c5e44a9f494f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:07.629648 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.629623 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-podinfo" pod "517ef3f2-04c5-4a53-8721-c5e44a9f494f" (UID: "517ef3f2-04c5-4a53-8721-c5e44a9f494f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:43:07.663068 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.663028 4904 generic.go:356] "Generic (PLEG): container finished" podID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerID="1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada" exitCode=0 Oct 30 19:43:07.663275 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.663245 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" Oct 30 19:43:07.663342 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.663265 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" event={"ID":"014713a3-4a5c-49ff-8417-be78ee3a1cc2","Type":"ContainerDied","Data":"1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada"} Oct 30 19:43:07.663342 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.663336 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8" event={"ID":"014713a3-4a5c-49ff-8417-be78ee3a1cc2","Type":"ContainerDied","Data":"45834312993f33db9d25523fcc5f1ca80eccbe8c82a1d6c5425587233e8944fb"} Oct 30 19:43:07.663455 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.663365 4904 scope.go:122] "RemoveContainer" containerID="1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada" Oct 30 19:43:07.667406 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.667379 4904 generic.go:356] "Generic (PLEG): container finished" podID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerID="53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5" exitCode=0 Oct 30 19:43:07.667542 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.667511 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" Oct 30 19:43:07.668060 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.667896 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" event={"ID":"47126122-ef5b-4e3f-9449-2fee7766f55a","Type":"ContainerDied","Data":"53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5"} Oct 30 19:43:07.668060 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.667956 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q" event={"ID":"47126122-ef5b-4e3f-9449-2fee7766f55a","Type":"ContainerDied","Data":"5ee4b6041bfc425c881d85477d79e5601a9a5e542f18d7ab18c1000f4be0c07a"} Oct 30 19:43:07.676350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.675854 4904 generic.go:356] "Generic (PLEG): container finished" podID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerID="849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf" exitCode=0 Oct 30 19:43:07.676350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.675947 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" event={"ID":"517ef3f2-04c5-4a53-8721-c5e44a9f494f","Type":"ContainerDied","Data":"849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf"} Oct 30 19:43:07.676350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.675984 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" event={"ID":"517ef3f2-04c5-4a53-8721-c5e44a9f494f","Type":"ContainerDied","Data":"51b4c6083b7b98b565cf657eee7f569c73580172209a8f28df761e1ac7a8f43d"} Oct 30 19:43:07.676350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.676054 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt" Oct 30 19:43:07.682260 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.682241 4904 scope.go:122] "RemoveContainer" containerID="1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada" Oct 30 19:43:07.682592 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:43:07.682570 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada\": container with ID starting with 1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada not found: ID does not exist" containerID="1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada" Oct 30 19:43:07.682691 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.682603 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada"} err="failed to get container status \"1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada\": rpc error: code = NotFound desc = could not find container \"1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada\": container with ID starting with 1c85716d52b355a5dfd162babf70a1c9555dd538c20e0bd71e84b97b8e4fcada not found: ID does not exist" Oct 30 19:43:07.682691 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.682624 4904 scope.go:122] "RemoveContainer" containerID="53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5" Oct 30 19:43:07.694017 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.693999 4904 scope.go:122] "RemoveContainer" containerID="53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5" Oct 30 19:43:07.694274 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:43:07.694257 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5\": container with ID starting with 53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5 not found: ID does not exist" containerID="53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5" Oct 30 19:43:07.694331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.694282 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5"} err="failed to get container status \"53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5\": rpc error: code = NotFound desc = could not find container \"53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5\": container with ID starting with 53ed6514fec377646572096caa135ff1d60b686d148f0c468490583fd10a0fe5 not found: ID does not exist" Oct 30 19:43:07.694331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.694298 4904 scope.go:122] "RemoveContainer" containerID="849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf" Oct 30 19:43:07.704872 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.704615 4904 scope.go:122] "RemoveContainer" containerID="849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf" Oct 30 19:43:07.705403 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:43:07.705380 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf\": container with ID starting with 849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf not found: ID does not exist" containerID="849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf" Oct 30 19:43:07.705555 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.705526 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf"} err="failed to get container status \"849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf\": rpc error: code = NotFound desc = could not find container \"849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf\": container with ID starting with 849a1fea481d4eb34db1409bb3e084017e746e4e430fd92ffe1072e5f09ed6cf not found: ID does not exist" Oct 30 19:43:07.721988 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.721957 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.722126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.721993 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.722126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.722008 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.722126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.722021 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lqz7\" (UniqueName: \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-kube-api-access-8lqz7\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.722126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.722035 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.722126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.722048 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.722126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.722059 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.722126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.722071 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.722126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.722083 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517ef3f2-04c5-4a53-8721-c5e44a9f494f-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:07.758319 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.758288 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q"] Oct 30 19:43:07.768178 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.766219 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4h44q"] Oct 30 19:43:07.797063 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.797038 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8"] Oct 30 19:43:07.802902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.802882 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-r8kg8"] Oct 30 19:43:07.861814 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.861779 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt"] Oct 30 19:43:07.864273 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:07.864242 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-4w5dt"] Oct 30 19:43:09.283216 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:09.283168 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" path="/var/lib/kubelet/pods/014713a3-4a5c-49ff-8417-be78ee3a1cc2/volumes" Oct 30 19:43:09.284847 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:09.284824 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" path="/var/lib/kubelet/pods/47126122-ef5b-4e3f-9449-2fee7766f55a/volumes" Oct 30 19:43:09.285693 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:09.285672 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" path="/var/lib/kubelet/pods/517ef3f2-04c5-4a53-8721-c5e44a9f494f/volumes" Oct 30 19:43:24.510546 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:43:24.510508 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:43:31.813818 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:31.813782 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6"] Oct 30 19:43:31.916918 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:31.916849 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" containerID="cri-o://2dc46a6bc7554e9c8c2ceefcc5c27cde062f44050727dfd30512c86d1bd9ea97" gracePeriod=30 Oct 30 19:43:35.324798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:35.324763 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg"] Oct 30 19:43:35.325301 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:35.325007 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" podUID="4da7f290-6c92-4372-9104-6b6530fcddb5" containerName="maas-api" containerID="cri-o://244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd" gracePeriod=30 Oct 30 19:43:35.833138 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:35.830080 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:43:35.874125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:35.873124 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-maas-api-tls\") pod \"4da7f290-6c92-4372-9104-6b6530fcddb5\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " Oct 30 19:43:35.874125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:35.873276 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4da7f290-6c92-4372-9104-6b6530fcddb5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4da7f290-6c92-4372-9104-6b6530fcddb5-openshift-service-ca\") pod \"4da7f290-6c92-4372-9104-6b6530fcddb5\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " Oct 30 19:43:35.874125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:35.873321 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da7f290-6c92-4372-9104-6b6530fcddb5-kube-api-access-phbwh\" (UniqueName: \"kubernetes.io/projected/4da7f290-6c92-4372-9104-6b6530fcddb5-kube-api-access-phbwh\") pod \"4da7f290-6c92-4372-9104-6b6530fcddb5\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " Oct 30 19:43:35.887283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:35.876279 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-metrics-certs\") pod \"4da7f290-6c92-4372-9104-6b6530fcddb5\" (UID: \"4da7f290-6c92-4372-9104-6b6530fcddb5\") " Oct 30 19:43:35.892122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:35.889141 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4da7f290-6c92-4372-9104-6b6530fcddb5-openshift-service-ca" pod "4da7f290-6c92-4372-9104-6b6530fcddb5" (UID: "4da7f290-6c92-4372-9104-6b6530fcddb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:36.008132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:35.986142 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4da7f290-6c92-4372-9104-6b6530fcddb5-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:36.056410 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.042911 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-metrics-certs" pod "4da7f290-6c92-4372-9104-6b6530fcddb5" (UID: "4da7f290-6c92-4372-9104-6b6530fcddb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:36.109873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.092902 4904 generic.go:356] "Generic (PLEG): container finished" podID="4da7f290-6c92-4372-9104-6b6530fcddb5" containerID="244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd" exitCode=0 Oct 30 19:43:36.109873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.093196 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" event={"ID":"4da7f290-6c92-4372-9104-6b6530fcddb5","Type":"ContainerDied","Data":"244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd"} Oct 30 19:43:36.109873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.093296 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" event={"ID":"4da7f290-6c92-4372-9104-6b6530fcddb5","Type":"ContainerDied","Data":"7af680fde6fa04ffe2697e97c867e1292f84d7a729490f924285ca16edbffc1b"} Oct 30 19:43:36.109873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.093334 4904 scope.go:122] "RemoveContainer" containerID="244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd" Oct 30 19:43:36.109873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.093525 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg" Oct 30 19:43:36.109873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.095714 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-maas-api-tls" pod "4da7f290-6c92-4372-9104-6b6530fcddb5" (UID: "4da7f290-6c92-4372-9104-6b6530fcddb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:36.109873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.097613 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da7f290-6c92-4372-9104-6b6530fcddb5-kube-api-access-phbwh" pod "4da7f290-6c92-4372-9104-6b6530fcddb5" (UID: "4da7f290-6c92-4372-9104-6b6530fcddb5"). InnerVolumeSpecName "kube-api-access-phbwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:36.109873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.108486 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:36.109873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.108518 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phbwh\" (UniqueName: \"kubernetes.io/projected/4da7f290-6c92-4372-9104-6b6530fcddb5-kube-api-access-phbwh\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:36.109873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.108534 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4da7f290-6c92-4372-9104-6b6530fcddb5-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:36.159025 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.157337 4904 scope.go:122] "RemoveContainer" containerID="244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd" Oct 30 19:43:36.164857 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:43:36.161262 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd\": container with ID starting with 244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd not found: ID does not exist" containerID="244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd" Oct 30 19:43:36.164857 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.161324 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd"} err="failed to get container status \"244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd\": rpc error: code = NotFound desc = could not find container \"244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd\": container with ID starting with 244dfcdd9c66f5496fbb3aa3f883c4f50738db7cdf3e396de760e9d59ddc92bd not found: ID does not exist" Oct 30 19:43:36.446881 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.446849 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg"] Oct 30 19:43:36.451368 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:36.451344 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7e89da-59ddbfc9b6-wgxsg"] Oct 30 19:43:37.107469 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.107432 4904 generic.go:356] "Generic (PLEG): container finished" podID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerID="2dc46a6bc7554e9c8c2ceefcc5c27cde062f44050727dfd30512c86d1bd9ea97" exitCode=0 Oct 30 19:43:37.107644 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.107625 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" event={"ID":"048a2238-713a-4aa8-a6b1-d49457c0e66a","Type":"ContainerDied","Data":"2dc46a6bc7554e9c8c2ceefcc5c27cde062f44050727dfd30512c86d1bd9ea97"} Oct 30 19:43:37.218906 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.218885 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:43:37.224166 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.224144 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4da7f290-6c92-4372-9104-6b6530fcddb5" path="/var/lib/kubelet/pods/4da7f290-6c92-4372-9104-6b6530fcddb5/volumes" Oct 30 19:43:37.224829 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.224813 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-data\") pod \"048a2238-713a-4aa8-a6b1-d49457c0e66a\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " Oct 30 19:43:37.224887 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.224863 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-credential-socket\") pod \"048a2238-713a-4aa8-a6b1-d49457c0e66a\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " Oct 30 19:43:37.224930 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.224920 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-envoy\") pod \"048a2238-713a-4aa8-a6b1-d49457c0e66a\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " Oct 30 19:43:37.224979 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.224958 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-kube-api-access-87prz\" (UniqueName: \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-kube-api-access-87prz\") pod \"048a2238-713a-4aa8-a6b1-d49457c0e66a\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " Oct 30 19:43:37.225080 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.225056 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-data" pod "048a2238-713a-4aa8-a6b1-d49457c0e66a" (UID: "048a2238-713a-4aa8-a6b1-d49457c0e66a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:37.225188 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.225172 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-credential-socket" pod "048a2238-713a-4aa8-a6b1-d49457c0e66a" (UID: "048a2238-713a-4aa8-a6b1-d49457c0e66a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:37.225265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.225250 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-socket\") pod \"048a2238-713a-4aa8-a6b1-d49457c0e66a\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " Oct 30 19:43:37.225321 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.225288 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-podinfo\") pod \"048a2238-713a-4aa8-a6b1-d49457c0e66a\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " Oct 30 19:43:37.225321 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.225310 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-certs\") pod \"048a2238-713a-4aa8-a6b1-d49457c0e66a\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " Oct 30 19:43:37.225409 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.225392 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048a2238-713a-4aa8-a6b1-d49457c0e66a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048a2238-713a-4aa8-a6b1-d49457c0e66a-istiod-ca-cert\") pod \"048a2238-713a-4aa8-a6b1-d49457c0e66a\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " Oct 30 19:43:37.225486 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.225471 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-token\" (UniqueName: \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-token\") pod \"048a2238-713a-4aa8-a6b1-d49457c0e66a\" (UID: \"048a2238-713a-4aa8-a6b1-d49457c0e66a\") " Oct 30 19:43:37.225622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.225573 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-socket" pod "048a2238-713a-4aa8-a6b1-d49457c0e66a" (UID: "048a2238-713a-4aa8-a6b1-d49457c0e66a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:37.225697 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.225635 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-certs" pod "048a2238-713a-4aa8-a6b1-d49457c0e66a" (UID: "048a2238-713a-4aa8-a6b1-d49457c0e66a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:37.226046 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.226018 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048a2238-713a-4aa8-a6b1-d49457c0e66a-istiod-ca-cert" pod "048a2238-713a-4aa8-a6b1-d49457c0e66a" (UID: "048a2238-713a-4aa8-a6b1-d49457c0e66a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:37.226629 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.226606 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048a2238-713a-4aa8-a6b1-d49457c0e66a-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:37.226729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.226636 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:37.226729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.226649 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:37.226729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.226662 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:37.226729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.226676 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:37.227441 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.227411 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-podinfo" pod "048a2238-713a-4aa8-a6b1-d49457c0e66a" (UID: "048a2238-713a-4aa8-a6b1-d49457c0e66a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:43:37.227694 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.227672 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-envoy" pod "048a2238-713a-4aa8-a6b1-d49457c0e66a" (UID: "048a2238-713a-4aa8-a6b1-d49457c0e66a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:37.228034 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.228013 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-token" pod "048a2238-713a-4aa8-a6b1-d49457c0e66a" (UID: "048a2238-713a-4aa8-a6b1-d49457c0e66a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:37.228084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.228043 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-kube-api-access-87prz" pod "048a2238-713a-4aa8-a6b1-d49457c0e66a" (UID: "048a2238-713a-4aa8-a6b1-d49457c0e66a"). InnerVolumeSpecName "kube-api-access-87prz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:37.328541 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.328513 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:37.328541 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.328538 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87prz\" (UniqueName: \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-kube-api-access-87prz\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:37.328541 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.328547 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:37.328740 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:37.328557 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048a2238-713a-4aa8-a6b1-d49457c0e66a-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:38.118451 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:38.118422 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" Oct 30 19:43:38.118451 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:38.118443 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6" event={"ID":"048a2238-713a-4aa8-a6b1-d49457c0e66a","Type":"ContainerDied","Data":"49a271541b6f1e200285ca0a491cff71a7bdfeece003248fb3ef9bdb8e452cbb"} Oct 30 19:43:38.119000 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:38.118509 4904 scope.go:122] "RemoveContainer" containerID="2dc46a6bc7554e9c8c2ceefcc5c27cde062f44050727dfd30512c86d1bd9ea97" Oct 30 19:43:38.203392 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:38.203359 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6"] Oct 30 19:43:38.214510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:38.214480 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-c8mr6"] Oct 30 19:43:39.226587 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:39.226553 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" path="/var/lib/kubelet/pods/048a2238-713a-4aa8-a6b1-d49457c0e66a/volumes" Oct 30 19:43:41.184931 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:41.184896 4904 scope.go:122] "RemoveContainer" containerID="330d1bafafd0b5c2b67525c0c5ad4683f0e8bc960927806c1958b95793638182" Oct 30 19:43:47.296733 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:47.296698 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275"] Oct 30 19:43:47.297155 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:47.296955 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" containerID="cri-o://d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196" gracePeriod=30 Oct 30 19:43:49.355370 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:49.355293 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx"] Oct 30 19:43:49.379264 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:49.379233 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4"] Oct 30 19:43:52.644904 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.644861 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:43:52.782657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.782617 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-data\") pod \"50014489-db75-4a42-a19d-f9a14b3f7571\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " Oct 30 19:43:52.782846 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.782727 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-kube-api-access-7tg86\" (UniqueName: \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-kube-api-access-7tg86\") pod \"50014489-db75-4a42-a19d-f9a14b3f7571\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " Oct 30 19:43:52.782846 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.782764 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50014489-db75-4a42-a19d-f9a14b3f7571-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50014489-db75-4a42-a19d-f9a14b3f7571-istio-podinfo\") pod \"50014489-db75-4a42-a19d-f9a14b3f7571\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " Oct 30 19:43:52.782846 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.782815 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-credential-socket\") pod \"50014489-db75-4a42-a19d-f9a14b3f7571\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " Oct 30 19:43:52.783033 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.782962 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-socket\") pod \"50014489-db75-4a42-a19d-f9a14b3f7571\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " Oct 30 19:43:52.783094 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.783052 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-certs\") pod \"50014489-db75-4a42-a19d-f9a14b3f7571\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " Oct 30 19:43:52.783204 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.783173 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-istio-token\" (UniqueName: \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-istio-token\") pod \"50014489-db75-4a42-a19d-f9a14b3f7571\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " Oct 30 19:43:52.783204 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.783200 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-envoy\") pod \"50014489-db75-4a42-a19d-f9a14b3f7571\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " Oct 30 19:43:52.783327 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.783245 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50014489-db75-4a42-a19d-f9a14b3f7571-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50014489-db75-4a42-a19d-f9a14b3f7571-istiod-ca-cert\") pod \"50014489-db75-4a42-a19d-f9a14b3f7571\" (UID: \"50014489-db75-4a42-a19d-f9a14b3f7571\") " Oct 30 19:43:52.784034 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.784008 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50014489-db75-4a42-a19d-f9a14b3f7571-istiod-ca-cert" pod "50014489-db75-4a42-a19d-f9a14b3f7571" (UID: "50014489-db75-4a42-a19d-f9a14b3f7571"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:52.784243 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.784226 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-data" pod "50014489-db75-4a42-a19d-f9a14b3f7571" (UID: "50014489-db75-4a42-a19d-f9a14b3f7571"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:52.784692 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.784644 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-socket" pod "50014489-db75-4a42-a19d-f9a14b3f7571" (UID: "50014489-db75-4a42-a19d-f9a14b3f7571"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:52.784883 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.784861 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-certs" pod "50014489-db75-4a42-a19d-f9a14b3f7571" (UID: "50014489-db75-4a42-a19d-f9a14b3f7571"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:52.785464 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.785435 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-credential-socket" pod "50014489-db75-4a42-a19d-f9a14b3f7571" (UID: "50014489-db75-4a42-a19d-f9a14b3f7571"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:52.787260 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.787233 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-envoy" pod "50014489-db75-4a42-a19d-f9a14b3f7571" (UID: "50014489-db75-4a42-a19d-f9a14b3f7571"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:52.787360 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.787278 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-kube-api-access-7tg86" pod "50014489-db75-4a42-a19d-f9a14b3f7571" (UID: "50014489-db75-4a42-a19d-f9a14b3f7571"). InnerVolumeSpecName "kube-api-access-7tg86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:52.787557 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.787527 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50014489-db75-4a42-a19d-f9a14b3f7571-istio-podinfo" pod "50014489-db75-4a42-a19d-f9a14b3f7571" (UID: "50014489-db75-4a42-a19d-f9a14b3f7571"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:43:52.788070 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.788041 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-istio-token" pod "50014489-db75-4a42-a19d-f9a14b3f7571" (UID: "50014489-db75-4a42-a19d-f9a14b3f7571"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:52.885688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.885573 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:52.885688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.885612 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tg86\" (UniqueName: \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-kube-api-access-7tg86\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:52.885688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.885631 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50014489-db75-4a42-a19d-f9a14b3f7571-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:52.885688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.885645 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:52.885688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.885660 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:52.885688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.885675 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:52.885688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.885688 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50014489-db75-4a42-a19d-f9a14b3f7571-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:52.886242 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.885703 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50014489-db75-4a42-a19d-f9a14b3f7571-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:52.886242 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:52.885716 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50014489-db75-4a42-a19d-f9a14b3f7571-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:43:53.467905 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:53.467868 4904 generic.go:356] "Generic (PLEG): container finished" podID="50014489-db75-4a42-a19d-f9a14b3f7571" containerID="d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196" exitCode=0 Oct 30 19:43:53.468275 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:53.468251 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" event={"ID":"50014489-db75-4a42-a19d-f9a14b3f7571","Type":"ContainerDied","Data":"d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196"} Oct 30 19:43:53.468437 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:53.468423 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" event={"ID":"50014489-db75-4a42-a19d-f9a14b3f7571","Type":"ContainerDied","Data":"443fb1842daa5a7986ba5d373a40ae7885e7e8ea48742cc03e56b89b4fcecf0c"} Oct 30 19:43:53.468541 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:53.468529 4904 scope.go:122] "RemoveContainer" containerID="d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196" Oct 30 19:43:53.468776 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:53.468762 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275" Oct 30 19:43:53.527313 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:53.521287 4904 scope.go:122] "RemoveContainer" containerID="d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196" Oct 30 19:43:53.527313 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:43:53.521725 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196\": container with ID starting with d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196 not found: ID does not exist" containerID="d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196" Oct 30 19:43:53.527313 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:53.521768 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196"} err="failed to get container status \"d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196\": rpc error: code = NotFound desc = could not find container \"d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196\": container with ID starting with d4fe46b174367cfb5d136c35ad1cbd77d3139b603af4071bbd712483c2e98196 not found: ID does not exist" Oct 30 19:43:53.704482 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:53.704446 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275"] Oct 30 19:43:54.088183 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:54.044508 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-bg275"] Oct 30 19:43:54.611577 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:43:54.611539 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:43:55.231493 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:43:55.231456 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" path="/var/lib/kubelet/pods/50014489-db75-4a42-a19d-f9a14b3f7571/volumes" Oct 30 19:44:04.378353 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.378292 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" containerID="cri-o://a4ba9b33930532748ae7ae6a607ee5e2daf0c034fcd0cd56fb7b0b9ba513a98a" gracePeriod=45 Oct 30 19:44:04.400809 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.400754 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" containerID="cri-o://798dd70f34b376b441f9e331c2d325efaedc72c6fd8c00e2a4a38e662d5b0d8a" gracePeriod=45 Oct 30 19:44:04.628703 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.628584 4904 generic.go:356] "Generic (PLEG): container finished" podID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerID="798dd70f34b376b441f9e331c2d325efaedc72c6fd8c00e2a4a38e662d5b0d8a" exitCode=0 Oct 30 19:44:04.629428 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.629359 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" event={"ID":"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed","Type":"ContainerDied","Data":"798dd70f34b376b441f9e331c2d325efaedc72c6fd8c00e2a4a38e662d5b0d8a"} Oct 30 19:44:04.633288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.633182 4904 generic.go:356] "Generic (PLEG): container finished" podID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerID="a4ba9b33930532748ae7ae6a607ee5e2daf0c034fcd0cd56fb7b0b9ba513a98a" exitCode=0 Oct 30 19:44:04.633288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.633234 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" event={"ID":"5c198194-5eff-4b2f-9847-ac737f4caaf6","Type":"ContainerDied","Data":"a4ba9b33930532748ae7ae6a607ee5e2daf0c034fcd0cd56fb7b0b9ba513a98a"} Oct 30 19:44:04.741238 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.741214 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:44:04.746813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.746785 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:44:04.887722 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.887628 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-home\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-home\") pod \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " Oct 30 19:44:04.887722 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.887681 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-kube-api-access-rj8pp\" (UniqueName: \"kubernetes.io/projected/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-kube-api-access-rj8pp\") pod \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " Oct 30 19:44:04.887904 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.887745 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tls-certs\" (UniqueName: \"kubernetes.io/secret/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tls-certs\") pod \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " Oct 30 19:44:04.887904 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.887814 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c198194-5eff-4b2f-9847-ac737f4caaf6-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c198194-5eff-4b2f-9847-ac737f4caaf6-tls-certs\") pod \"5c198194-5eff-4b2f-9847-ac737f4caaf6\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " Oct 30 19:44:04.887904 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.887853 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-home\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-home\") pod \"5c198194-5eff-4b2f-9847-ac737f4caaf6\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " Oct 30 19:44:04.888913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.888197 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-home" pod "5c198194-5eff-4b2f-9847-ac737f4caaf6" (UID: "5c198194-5eff-4b2f-9847-ac737f4caaf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.888913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.888242 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c198194-5eff-4b2f-9847-ac737f4caaf6-kube-api-access-kk6x2\" (UniqueName: \"kubernetes.io/projected/5c198194-5eff-4b2f-9847-ac737f4caaf6-kube-api-access-kk6x2\") pod \"5c198194-5eff-4b2f-9847-ac737f4caaf6\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " Oct 30 19:44:04.888913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.888332 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-dshm\") pod \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " Oct 30 19:44:04.888913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.888462 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-model-cache\") pod \"5c198194-5eff-4b2f-9847-ac737f4caaf6\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " Oct 30 19:44:04.888913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.888504 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-dshm\") pod \"5c198194-5eff-4b2f-9847-ac737f4caaf6\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " Oct 30 19:44:04.888913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.888554 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-home" pod "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" (UID: "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.888913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.888587 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-model-cache\") pod \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " Oct 30 19:44:04.888913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.888651 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tmp-dir\") pod \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\" (UID: \"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed\") " Oct 30 19:44:04.888913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.888815 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-tmp-dir\") pod \"5c198194-5eff-4b2f-9847-ac737f4caaf6\" (UID: \"5c198194-5eff-4b2f-9847-ac737f4caaf6\") " Oct 30 19:44:04.889525 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.888919 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-model-cache" pod "5c198194-5eff-4b2f-9847-ac737f4caaf6" (UID: "5c198194-5eff-4b2f-9847-ac737f4caaf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.889525 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.889163 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-model-cache" pod "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" (UID: "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.889807 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.889748 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.889807 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.889775 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.889807 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.889788 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.889807 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.889803 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.890080 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.890004 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-tmp-dir" pod "5c198194-5eff-4b2f-9847-ac737f4caaf6" (UID: "5c198194-5eff-4b2f-9847-ac737f4caaf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.893194 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.890561 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tmp-dir" pod "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" (UID: "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.893194 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.892908 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-dshm" pod "5c198194-5eff-4b2f-9847-ac737f4caaf6" (UID: "5c198194-5eff-4b2f-9847-ac737f4caaf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.893541 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.893518 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-dshm" pod "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" (UID: "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.894478 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.894426 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tls-certs" pod "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" (UID: "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:04.896228 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.896199 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c198194-5eff-4b2f-9847-ac737f4caaf6-kube-api-access-kk6x2" pod "5c198194-5eff-4b2f-9847-ac737f4caaf6" (UID: "5c198194-5eff-4b2f-9847-ac737f4caaf6"). InnerVolumeSpecName "kube-api-access-kk6x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:04.896404 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.896381 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c198194-5eff-4b2f-9847-ac737f4caaf6-tls-certs" pod "5c198194-5eff-4b2f-9847-ac737f4caaf6" (UID: "5c198194-5eff-4b2f-9847-ac737f4caaf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:04.897859 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.897829 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-kube-api-access-rj8pp" pod "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" (UID: "c111f7d7-b7f6-4ed2-8087-e4fe9d307eed"). InnerVolumeSpecName "kube-api-access-rj8pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:04.991091 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.991054 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.991091 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.991095 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj8pp\" (UniqueName: \"kubernetes.io/projected/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-kube-api-access-rj8pp\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.991374 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.991150 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.991374 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.991164 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c198194-5eff-4b2f-9847-ac737f4caaf6-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.991374 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.991177 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk6x2\" (UniqueName: \"kubernetes.io/projected/5c198194-5eff-4b2f-9847-ac737f4caaf6-kube-api-access-kk6x2\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.991374 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.991189 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.991374 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.991201 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c198194-5eff-4b2f-9847-ac737f4caaf6-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:04.991374 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:04.991213 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:05.584660 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.584575 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 19:44:05.642840 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.642810 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" Oct 30 19:44:05.642840 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.642824 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx" event={"ID":"5c198194-5eff-4b2f-9847-ac737f4caaf6","Type":"ContainerDied","Data":"12f6365db51aba4a7fbb92977e77f8a56f06c9b64bdba2270e17864a043d0d23"} Oct 30 19:44:05.643064 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.642876 4904 scope.go:122] "RemoveContainer" containerID="a4ba9b33930532748ae7ae6a607ee5e2daf0c034fcd0cd56fb7b0b9ba513a98a" Oct 30 19:44:05.645230 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.644575 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" Oct 30 19:44:05.645230 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.644604 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4" event={"ID":"c111f7d7-b7f6-4ed2-8087-e4fe9d307eed","Type":"ContainerDied","Data":"5f6a98be845c6a3d4686d6cd8dbc5326eb8fe7b6a308ef6333f29e83abcd53a1"} Oct 30 19:44:05.657204 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.657183 4904 scope.go:122] "RemoveContainer" containerID="798dd70f34b376b441f9e331c2d325efaedc72c6fd8c00e2a4a38e662d5b0d8a" Oct 30 19:44:05.702858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.702836 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx"] Oct 30 19:44:05.716062 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.716038 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-2-w3-dd23d1-kserve-676568fb49-6skzx"] Oct 30 19:44:05.747616 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.747570 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4"] Oct 30 19:44:05.751551 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:05.751529 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/premium-w3-dd23d1-kserve-7d7794cc95-6hmt4"] Oct 30 19:44:07.237269 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:07.237239 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" path="/var/lib/kubelet/pods/5c198194-5eff-4b2f-9847-ac737f4caaf6/volumes" Oct 30 19:44:07.237687 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:07.237672 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" path="/var/lib/kubelet/pods/c111f7d7-b7f6-4ed2-8087-e4fe9d307eed/volumes" Oct 30 19:44:24.515302 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:44:24.515259 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:44:33.025129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:33.015374 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v"] Oct 30 19:44:33.025129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:33.015640 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" podUID="22d87143-d4c2-483a-adc6-e79c448a708b" containerName="maas-api" containerID="cri-o://f3f5650df557b373ce8b91285a4d01e7872c32a14996666ae720ffe8345a3c51" gracePeriod=30 Oct 30 19:44:33.282149 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:33.257199 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x4dsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:33.282149 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:33.257269 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x4dsn" podUID="2c012900-326d-42d5-8076-3bbbdbb1e852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:33.834177 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:33.829325 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ggdsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:33.834177 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:33.829425 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" podUID="0c8a2a4b-6240-40fa-a873-d2300e317a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:34.121751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.120192 4904 generic.go:356] "Generic (PLEG): container finished" podID="22d87143-d4c2-483a-adc6-e79c448a708b" containerID="f3f5650df557b373ce8b91285a4d01e7872c32a14996666ae720ffe8345a3c51" exitCode=0 Oct 30 19:44:34.121751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.120517 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" event={"ID":"22d87143-d4c2-483a-adc6-e79c448a708b","Type":"ContainerDied","Data":"f3f5650df557b373ce8b91285a4d01e7872c32a14996666ae720ffe8345a3c51"} Oct 30 19:44:34.190055 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.190030 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:44:34.315942 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.315906 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d87143-d4c2-483a-adc6-e79c448a708b-kube-api-access-rm6vp\" (UniqueName: \"kubernetes.io/projected/22d87143-d4c2-483a-adc6-e79c448a708b-kube-api-access-rm6vp\") pod \"22d87143-d4c2-483a-adc6-e79c448a708b\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " Oct 30 19:44:34.316170 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.316005 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d87143-d4c2-483a-adc6-e79c448a708b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d87143-d4c2-483a-adc6-e79c448a708b-openshift-service-ca\") pod \"22d87143-d4c2-483a-adc6-e79c448a708b\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " Oct 30 19:44:34.316299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.316278 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-maas-api-tls\") pod \"22d87143-d4c2-483a-adc6-e79c448a708b\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " Oct 30 19:44:34.316392 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.316368 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-metrics-certs\") pod \"22d87143-d4c2-483a-adc6-e79c448a708b\" (UID: \"22d87143-d4c2-483a-adc6-e79c448a708b\") " Oct 30 19:44:34.316476 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.316447 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d87143-d4c2-483a-adc6-e79c448a708b-openshift-service-ca" pod "22d87143-d4c2-483a-adc6-e79c448a708b" (UID: "22d87143-d4c2-483a-adc6-e79c448a708b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:34.317030 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.316997 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d87143-d4c2-483a-adc6-e79c448a708b-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.318938 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.318907 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-metrics-certs" pod "22d87143-d4c2-483a-adc6-e79c448a708b" (UID: "22d87143-d4c2-483a-adc6-e79c448a708b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:34.319173 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.319151 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d87143-d4c2-483a-adc6-e79c448a708b-kube-api-access-rm6vp" pod "22d87143-d4c2-483a-adc6-e79c448a708b" (UID: "22d87143-d4c2-483a-adc6-e79c448a708b"). InnerVolumeSpecName "kube-api-access-rm6vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:34.319335 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.319305 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-maas-api-tls" pod "22d87143-d4c2-483a-adc6-e79c448a708b" (UID: "22d87143-d4c2-483a-adc6-e79c448a708b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:34.419682 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.419545 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.419961 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.419933 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d87143-d4c2-483a-adc6-e79c448a708b-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.420156 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:34.419968 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm6vp\" (UniqueName: \"kubernetes.io/projected/22d87143-d4c2-483a-adc6-e79c448a708b-kube-api-access-rm6vp\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:35.130888 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:35.130857 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" Oct 30 19:44:35.131398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:35.130888 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v" event={"ID":"22d87143-d4c2-483a-adc6-e79c448a708b","Type":"ContainerDied","Data":"8ff1bc16ff9b0ef01006e5d159bd904b93e039a51ebd56c7d6dbbdbfd1a91252"} Oct 30 19:44:35.131398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:35.130955 4904 scope.go:122] "RemoveContainer" containerID="f3f5650df557b373ce8b91285a4d01e7872c32a14996666ae720ffe8345a3c51" Oct 30 19:44:35.210402 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:35.210349 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v"] Oct 30 19:44:35.254726 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:35.254686 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a4e2f-6cff747445-zjk4v"] Oct 30 19:44:37.224195 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:37.224161 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d87143-d4c2-483a-adc6-e79c448a708b" path="/var/lib/kubelet/pods/22d87143-d4c2-483a-adc6-e79c448a708b/volumes" Oct 30 19:44:54.512415 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:44:54.512379 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:44:57.331473 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:57.331441 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq"] Oct 30 19:44:57.332074 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:57.331693 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" podUID="b68a0ff5-8958-4860-bcba-71f0e4a89d7d" containerName="maas-api" containerID="cri-o://16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1" gracePeriod=30 Oct 30 19:44:58.037970 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.037652 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:44:58.144748 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.144711 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-maas-api-tls\") pod \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " Oct 30 19:44:58.144905 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.144875 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-openshift-service-ca\") pod \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " Oct 30 19:44:58.144958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.144944 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-kube-api-access-58h2n\" (UniqueName: \"kubernetes.io/projected/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-kube-api-access-58h2n\") pod \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " Oct 30 19:44:58.145023 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.145009 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-metrics-certs\") pod \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\" (UID: \"b68a0ff5-8958-4860-bcba-71f0e4a89d7d\") " Oct 30 19:44:58.145341 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.145310 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-openshift-service-ca" pod "b68a0ff5-8958-4860-bcba-71f0e4a89d7d" (UID: "b68a0ff5-8958-4860-bcba-71f0e4a89d7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:58.145787 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.145760 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:58.147203 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.147173 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-metrics-certs" pod "b68a0ff5-8958-4860-bcba-71f0e4a89d7d" (UID: "b68a0ff5-8958-4860-bcba-71f0e4a89d7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:58.147524 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.147495 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-maas-api-tls" pod "b68a0ff5-8958-4860-bcba-71f0e4a89d7d" (UID: "b68a0ff5-8958-4860-bcba-71f0e4a89d7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:58.147629 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.147583 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-kube-api-access-58h2n" pod "b68a0ff5-8958-4860-bcba-71f0e4a89d7d" (UID: "b68a0ff5-8958-4860-bcba-71f0e4a89d7d"). InnerVolumeSpecName "kube-api-access-58h2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:58.247353 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.247283 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:58.247353 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.247318 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58h2n\" (UniqueName: \"kubernetes.io/projected/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-kube-api-access-58h2n\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:58.247353 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.247332 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68a0ff5-8958-4860-bcba-71f0e4a89d7d-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:44:58.348927 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.348884 4904 generic.go:356] "Generic (PLEG): container finished" podID="b68a0ff5-8958-4860-bcba-71f0e4a89d7d" containerID="16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1" exitCode=0 Oct 30 19:44:58.349381 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.348942 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" Oct 30 19:44:58.349381 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.348984 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" event={"ID":"b68a0ff5-8958-4860-bcba-71f0e4a89d7d","Type":"ContainerDied","Data":"16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1"} Oct 30 19:44:58.349381 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.349048 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq" event={"ID":"b68a0ff5-8958-4860-bcba-71f0e4a89d7d","Type":"ContainerDied","Data":"8adbc804cc7f65f1d725623ee6004739ec66e8060888c1b690b79c373ca4b558"} Oct 30 19:44:58.349381 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.349077 4904 scope.go:122] "RemoveContainer" containerID="16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1" Oct 30 19:44:58.371697 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.371659 4904 scope.go:122] "RemoveContainer" containerID="16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1" Oct 30 19:44:58.372180 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:44:58.372154 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1\": container with ID starting with 16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1 not found: ID does not exist" containerID="16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1" Oct 30 19:44:58.372312 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.372189 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1"} err="failed to get container status \"16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1\": rpc error: code = NotFound desc = could not find container \"16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1\": container with ID starting with 16fc3c95a8b5bbbd332c3e71831a5d63d59af6117214f8056582260d6ad90ba1 not found: ID does not exist" Oct 30 19:44:58.421087 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.421046 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq"] Oct 30 19:44:58.427899 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:58.427875 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67953efd-956bfdb9-z49lq"] Oct 30 19:44:59.224558 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:44:59.224518 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68a0ff5-8958-4860-bcba-71f0e4a89d7d" path="/var/lib/kubelet/pods/b68a0ff5-8958-4860-bcba-71f0e4a89d7d/volumes" Oct 30 19:45:00.239127 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.238166 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z"] Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239185 4904 cpu_manager.go:414] "containerMap: removing container" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239206 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239221 4904 cpu_manager.go:414] "containerMap: removing container" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239229 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239248 4904 cpu_manager.go:414] "containerMap: removing container" podUID="4da7f290-6c92-4372-9104-6b6530fcddb5" containerName="maas-api" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239255 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da7f290-6c92-4372-9104-6b6530fcddb5" containerName="maas-api" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239281 4904 cpu_manager.go:414] "containerMap: removing container" podUID="b68a0ff5-8958-4860-bcba-71f0e4a89d7d" containerName="maas-api" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239288 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68a0ff5-8958-4860-bcba-71f0e4a89d7d" containerName="maas-api" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239315 4904 cpu_manager.go:414] "containerMap: removing container" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239323 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239346 4904 cpu_manager.go:414] "containerMap: removing container" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239353 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239362 4904 cpu_manager.go:414] "containerMap: removing container" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239370 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239382 4904 cpu_manager.go:414] "containerMap: removing container" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239388 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239401 4904 cpu_manager.go:414] "containerMap: removing container" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239410 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239434 4904 cpu_manager.go:414] "containerMap: removing container" podUID="22d87143-d4c2-483a-adc6-e79c448a708b" containerName="maas-api" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239441 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d87143-d4c2-483a-adc6-e79c448a708b" containerName="maas-api" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239595 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c198194-5eff-4b2f-9847-ac737f4caaf6" containerName="main" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239605 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="014713a3-4a5c-49ff-8417-be78ee3a1cc2" containerName="istio-proxy" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239618 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da7f290-6c92-4372-9104-6b6530fcddb5" containerName="maas-api" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239631 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68a0ff5-8958-4860-bcba-71f0e4a89d7d" containerName="maas-api" Oct 30 19:45:00.239630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239641 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="47126122-ef5b-4e3f-9449-2fee7766f55a" containerName="istio-proxy" Oct 30 19:45:00.254798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239651 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="50014489-db75-4a42-a19d-f9a14b3f7571" containerName="istio-proxy" Oct 30 19:45:00.254798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239658 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="048a2238-713a-4aa8-a6b1-d49457c0e66a" containerName="istio-proxy" Oct 30 19:45:00.254798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239668 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="c111f7d7-b7f6-4ed2-8087-e4fe9d307eed" containerName="main" Oct 30 19:45:00.254798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239676 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="517ef3f2-04c5-4a53-8721-c5e44a9f494f" containerName="istio-proxy" Oct 30 19:45:00.254798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.239685 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d87143-d4c2-483a-adc6-e79c448a708b" containerName="maas-api" Oct 30 19:45:00.254798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.252410 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" Oct 30 19:45:00.282379 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.279407 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4qb5\"" Oct 30 19:45:00.303143 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.300562 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngfhr\" (UniqueName: \"kubernetes.io/projected/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-kube-api-access-ngfhr\") pod \"maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z\" (UID: \"05fe0b01-aaa5-4c59-9848-48c831f7dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" Oct 30 19:45:00.303143 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.300834 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z\" (UID: \"05fe0b01-aaa5-4c59-9848-48c831f7dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" Oct 30 19:45:00.382139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.382087 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl"] Oct 30 19:45:00.388775 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.388749 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" Oct 30 19:45:00.402056 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.402027 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p"] Oct 30 19:45:00.402974 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.402945 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z\" (UID: \"05fe0b01-aaa5-4c59-9848-48c831f7dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" Oct 30 19:45:00.403313 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.403293 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfhr\" (UniqueName: \"kubernetes.io/projected/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-kube-api-access-ngfhr\") pod \"maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z\" (UID: \"05fe0b01-aaa5-4c59-9848-48c831f7dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" Oct 30 19:45:00.404498 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.404475 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z\" (UID: \"05fe0b01-aaa5-4c59-9848-48c831f7dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" Oct 30 19:45:00.407336 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.407174 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" Oct 30 19:45:00.409466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.409444 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z"] Oct 30 19:45:00.421122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.417940 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl"] Oct 30 19:45:00.428364 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.425623 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7"] Oct 30 19:45:00.433342 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.433298 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngfhr\" (UniqueName: \"kubernetes.io/projected/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-kube-api-access-ngfhr\") pod \"maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z\" (UID: \"05fe0b01-aaa5-4c59-9848-48c831f7dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" Oct 30 19:45:00.454125 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.447258 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" Oct 30 19:45:00.477132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.459743 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p"] Oct 30 19:45:00.477132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.469488 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7"] Oct 30 19:45:00.643925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.641925 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlnqr\" (UniqueName: \"kubernetes.io/projected/c1d48683-e2d8-4332-acb2-abd4133fff88-kube-api-access-hlnqr\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p\" (UID: \"c1d48683-e2d8-4332-acb2-abd4133fff88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" Oct 30 19:45:00.643925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.642072 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca5963c-22d7-4688-b367-1aa657eb37b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl\" (UID: \"7ca5963c-22d7-4688-b367-1aa657eb37b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" Oct 30 19:45:00.643925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.642850 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d48683-e2d8-4332-acb2-abd4133fff88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p\" (UID: \"c1d48683-e2d8-4332-acb2-abd4133fff88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" Oct 30 19:45:00.643925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.643118 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" Oct 30 19:45:00.643925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.643516 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5pvt\" (UniqueName: \"kubernetes.io/projected/7ca5963c-22d7-4688-b367-1aa657eb37b4-kube-api-access-z5pvt\") pod \"maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl\" (UID: \"7ca5963c-22d7-4688-b367-1aa657eb37b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" Oct 30 19:45:00.643925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.643568 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkb99\" (UniqueName: \"kubernetes.io/projected/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-kube-api-access-kkb99\") pod \"maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7\" (UID: \"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" Oct 30 19:45:00.643925 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.643635 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7\" (UID: \"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" Oct 30 19:45:00.747398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.745545 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5pvt\" (UniqueName: \"kubernetes.io/projected/7ca5963c-22d7-4688-b367-1aa657eb37b4-kube-api-access-z5pvt\") pod \"maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl\" (UID: \"7ca5963c-22d7-4688-b367-1aa657eb37b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" Oct 30 19:45:00.747398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.745606 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkb99\" (UniqueName: \"kubernetes.io/projected/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-kube-api-access-kkb99\") pod \"maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7\" (UID: \"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" Oct 30 19:45:00.747398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.745666 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7\" (UID: \"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" Oct 30 19:45:00.747398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.745819 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlnqr\" (UniqueName: \"kubernetes.io/projected/c1d48683-e2d8-4332-acb2-abd4133fff88-kube-api-access-hlnqr\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p\" (UID: \"c1d48683-e2d8-4332-acb2-abd4133fff88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" Oct 30 19:45:00.747398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.745919 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca5963c-22d7-4688-b367-1aa657eb37b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl\" (UID: \"7ca5963c-22d7-4688-b367-1aa657eb37b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" Oct 30 19:45:00.747398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.745949 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d48683-e2d8-4332-acb2-abd4133fff88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p\" (UID: \"c1d48683-e2d8-4332-acb2-abd4133fff88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" Oct 30 19:45:00.747398 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.746937 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d48683-e2d8-4332-acb2-abd4133fff88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p\" (UID: \"c1d48683-e2d8-4332-acb2-abd4133fff88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" Oct 30 19:45:00.747869 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.747766 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7\" (UID: \"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" Oct 30 19:45:00.748168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.748144 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca5963c-22d7-4688-b367-1aa657eb37b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl\" (UID: \"7ca5963c-22d7-4688-b367-1aa657eb37b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" Oct 30 19:45:00.776783 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.776761 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlnqr\" (UniqueName: \"kubernetes.io/projected/c1d48683-e2d8-4332-acb2-abd4133fff88-kube-api-access-hlnqr\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p\" (UID: \"c1d48683-e2d8-4332-acb2-abd4133fff88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" Oct 30 19:45:00.780040 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.780019 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkb99\" (UniqueName: \"kubernetes.io/projected/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-kube-api-access-kkb99\") pod \"maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7\" (UID: \"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" Oct 30 19:45:00.780240 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.780217 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5pvt\" (UniqueName: \"kubernetes.io/projected/7ca5963c-22d7-4688-b367-1aa657eb37b4-kube-api-access-z5pvt\") pod \"maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl\" (UID: \"7ca5963c-22d7-4688-b367-1aa657eb37b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" Oct 30 19:45:00.928343 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:00.928305 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" Oct 30 19:45:01.014517 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:01.013790 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z"] Oct 30 19:45:01.057418 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:01.057296 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" Oct 30 19:45:01.078024 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:01.078000 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" Oct 30 19:45:01.250923 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:01.250844 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7"] Oct 30 19:45:01.257841 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:45:01.257808 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b2d89e_87fd_4ffc_a8f0_7dadfc414fb1.slice/crio-882081dd9ab4b27b853ca33e766df051247e7f9b2c2ad309695e712eda737d7e WatchSource:0}: Error finding container 882081dd9ab4b27b853ca33e766df051247e7f9b2c2ad309695e712eda737d7e: Status 404 returned error can't find the container with id 882081dd9ab4b27b853ca33e766df051247e7f9b2c2ad309695e712eda737d7e Oct 30 19:45:01.397239 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:45:01.397168 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca5963c_22d7_4688_b367_1aa657eb37b4.slice/crio-a9e5805e278f1959775f8ff01cbffc2f33366926e806d94e8f338cbf4b031a42 WatchSource:0}: Error finding container a9e5805e278f1959775f8ff01cbffc2f33366926e806d94e8f338cbf4b031a42: Status 404 returned error can't find the container with id a9e5805e278f1959775f8ff01cbffc2f33366926e806d94e8f338cbf4b031a42 Oct 30 19:45:01.397508 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:01.397486 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl"] Oct 30 19:45:01.401630 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:01.401608 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p"] Oct 30 19:45:01.412742 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:45:01.412711 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d48683_e2d8_4332_acb2_abd4133fff88.slice/crio-fc3d94da4ee1069db6b7f0048b06161bb3ff39755ec6c2d7083647ca364b90cb WatchSource:0}: Error finding container fc3d94da4ee1069db6b7f0048b06161bb3ff39755ec6c2d7083647ca364b90cb: Status 404 returned error can't find the container with id fc3d94da4ee1069db6b7f0048b06161bb3ff39755ec6c2d7083647ca364b90cb Oct 30 19:45:01.422947 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:01.422564 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" event={"ID":"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1","Type":"ContainerStarted","Data":"882081dd9ab4b27b853ca33e766df051247e7f9b2c2ad309695e712eda737d7e"} Oct 30 19:45:01.429061 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:01.429039 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" event={"ID":"05fe0b01-aaa5-4c59-9848-48c831f7dc4d","Type":"ContainerStarted","Data":"ab74474393ca16a984856e2b6e5db4c8f3b7d2c4d15347daebb5543fbf547ca2"} Oct 30 19:45:01.430504 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:01.430476 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" event={"ID":"7ca5963c-22d7-4688-b367-1aa657eb37b4","Type":"ContainerStarted","Data":"a9e5805e278f1959775f8ff01cbffc2f33366926e806d94e8f338cbf4b031a42"} Oct 30 19:45:02.441753 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.441724 4904 generic.go:356] "Generic (PLEG): container finished" podID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerID="4c6c2fb2aa75a94d8a196cb6fe3cbe0b902f840ba81c904364ce8c2fd2e25ccb" exitCode=60 Oct 30 19:45:02.442180 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.441807 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" event={"ID":"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1","Type":"ContainerDied","Data":"4c6c2fb2aa75a94d8a196cb6fe3cbe0b902f840ba81c904364ce8c2fd2e25ccb"} Oct 30 19:45:02.442437 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.442419 4904 scope.go:122] "RemoveContainer" containerID="4c6c2fb2aa75a94d8a196cb6fe3cbe0b902f840ba81c904364ce8c2fd2e25ccb" Oct 30 19:45:02.443884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.443858 4904 generic.go:356] "Generic (PLEG): container finished" podID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerID="50cbc66691057e2354275f77a2a7604dc54e08e46342a155c55b977a610fe21d" exitCode=60 Oct 30 19:45:02.444176 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.444153 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" event={"ID":"05fe0b01-aaa5-4c59-9848-48c831f7dc4d","Type":"ContainerDied","Data":"50cbc66691057e2354275f77a2a7604dc54e08e46342a155c55b977a610fe21d"} Oct 30 19:45:02.444287 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.444271 4904 scope.go:122] "RemoveContainer" containerID="50cbc66691057e2354275f77a2a7604dc54e08e46342a155c55b977a610fe21d" Oct 30 19:45:02.449773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.449752 4904 generic.go:356] "Generic (PLEG): container finished" podID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerID="809a89238b9c39ae9108818881b14cfb0ca85b94613e8e9b0bfb418239170b41" exitCode=60 Oct 30 19:45:02.449869 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.449844 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" event={"ID":"c1d48683-e2d8-4332-acb2-abd4133fff88","Type":"ContainerDied","Data":"809a89238b9c39ae9108818881b14cfb0ca85b94613e8e9b0bfb418239170b41"} Oct 30 19:45:02.449930 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.449904 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" event={"ID":"c1d48683-e2d8-4332-acb2-abd4133fff88","Type":"ContainerStarted","Data":"fc3d94da4ee1069db6b7f0048b06161bb3ff39755ec6c2d7083647ca364b90cb"} Oct 30 19:45:02.450090 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.450067 4904 scope.go:122] "RemoveContainer" containerID="809a89238b9c39ae9108818881b14cfb0ca85b94613e8e9b0bfb418239170b41" Oct 30 19:45:02.451497 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.451475 4904 generic.go:356] "Generic (PLEG): container finished" podID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerID="7bbd662eba7cde8eb27a51c8729438a358775494b556489bbab76a326b14b04d" exitCode=60 Oct 30 19:45:02.451594 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.451562 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" event={"ID":"7ca5963c-22d7-4688-b367-1aa657eb37b4","Type":"ContainerDied","Data":"7bbd662eba7cde8eb27a51c8729438a358775494b556489bbab76a326b14b04d"} Oct 30 19:45:02.451751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:02.451736 4904 scope.go:122] "RemoveContainer" containerID="7bbd662eba7cde8eb27a51c8729438a358775494b556489bbab76a326b14b04d" Oct 30 19:45:03.462028 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:03.461988 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" event={"ID":"c1d48683-e2d8-4332-acb2-abd4133fff88","Type":"ContainerStarted","Data":"e22ffdf8ca2cdf12035dd3cc13c435e298a27ab233cbfc5c112a1eb002589f3f"} Oct 30 19:45:03.464907 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:03.464872 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" event={"ID":"7ca5963c-22d7-4688-b367-1aa657eb37b4","Type":"ContainerStarted","Data":"b7aad3aa366c0ebc07dc22f91dabc248d34a7092c9159ee5465ba2d1ae1baef3"} Oct 30 19:45:03.467075 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:03.467052 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" event={"ID":"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1","Type":"ContainerStarted","Data":"998f20c91fa81c457cf8d8cca30da06014f257847fe48a8642086f4db0a98f1e"} Oct 30 19:45:03.469017 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:03.468994 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" event={"ID":"05fe0b01-aaa5-4c59-9848-48c831f7dc4d","Type":"ContainerStarted","Data":"a2df84e3e0f54ce168465b568f0c8e6be3bc0f22bde6f08351be6a1884e9f60a"} Oct 30 19:45:03.537892 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:03.537849 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" podStartSLOduration=3.537833179 podStartE2EDuration="3.537833179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.534633761 +0000 UTC m=+1464.448238379" watchObservedRunningTime="2026-10-30 19:45:03.537833179 +0000 UTC m=+1464.451437799" Oct 30 19:45:03.594462 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:03.594425 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" podStartSLOduration=3.594411356 podStartE2EDuration="3.594411356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.589211517 +0000 UTC m=+1464.502816133" watchObservedRunningTime="2026-10-30 19:45:03.594411356 +0000 UTC m=+1464.508015973" Oct 30 19:45:03.644469 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:03.644426 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" podStartSLOduration=3.644412517 podStartE2EDuration="3.644412517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.64168232 +0000 UTC m=+1464.555286937" watchObservedRunningTime="2026-10-30 19:45:03.644412517 +0000 UTC m=+1464.558017178" Oct 30 19:45:03.691405 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:03.691364 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" podStartSLOduration=3.691348499 podStartE2EDuration="3.691348499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.687360293 +0000 UTC m=+1464.600964909" watchObservedRunningTime="2026-10-30 19:45:03.691348499 +0000 UTC m=+1464.604953118" Oct 30 19:45:04.479008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.478972 4904 generic.go:356] "Generic (PLEG): container finished" podID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerID="998f20c91fa81c457cf8d8cca30da06014f257847fe48a8642086f4db0a98f1e" exitCode=60 Oct 30 19:45:04.479431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.479056 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" event={"ID":"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1","Type":"ContainerDied","Data":"998f20c91fa81c457cf8d8cca30da06014f257847fe48a8642086f4db0a98f1e"} Oct 30 19:45:04.479431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.479162 4904 scope.go:122] "RemoveContainer" containerID="4c6c2fb2aa75a94d8a196cb6fe3cbe0b902f840ba81c904364ce8c2fd2e25ccb" Oct 30 19:45:04.479431 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.479345 4904 scope.go:122] "RemoveContainer" containerID="998f20c91fa81c457cf8d8cca30da06014f257847fe48a8642086f4db0a98f1e" Oct 30 19:45:04.479679 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:04.479653 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7_odh-ai-gateway-infra(74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" Oct 30 19:45:04.481181 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.481154 4904 generic.go:356] "Generic (PLEG): container finished" podID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerID="a2df84e3e0f54ce168465b568f0c8e6be3bc0f22bde6f08351be6a1884e9f60a" exitCode=60 Oct 30 19:45:04.481285 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.481215 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" event={"ID":"05fe0b01-aaa5-4c59-9848-48c831f7dc4d","Type":"ContainerDied","Data":"a2df84e3e0f54ce168465b568f0c8e6be3bc0f22bde6f08351be6a1884e9f60a"} Oct 30 19:45:04.481507 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.481491 4904 scope.go:122] "RemoveContainer" containerID="a2df84e3e0f54ce168465b568f0c8e6be3bc0f22bde6f08351be6a1884e9f60a" Oct 30 19:45:04.481811 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:04.481789 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z_odh-ai-gateway-infra(05fe0b01-aaa5-4c59-9848-48c831f7dc4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" Oct 30 19:45:04.483237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.483213 4904 generic.go:356] "Generic (PLEG): container finished" podID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerID="e22ffdf8ca2cdf12035dd3cc13c435e298a27ab233cbfc5c112a1eb002589f3f" exitCode=60 Oct 30 19:45:04.483341 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.483280 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" event={"ID":"c1d48683-e2d8-4332-acb2-abd4133fff88","Type":"ContainerDied","Data":"e22ffdf8ca2cdf12035dd3cc13c435e298a27ab233cbfc5c112a1eb002589f3f"} Oct 30 19:45:04.483533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.483519 4904 scope.go:122] "RemoveContainer" containerID="e22ffdf8ca2cdf12035dd3cc13c435e298a27ab233cbfc5c112a1eb002589f3f" Oct 30 19:45:04.483797 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:04.483778 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p_odh-ai-gateway-infra(c1d48683-e2d8-4332-acb2-abd4133fff88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" Oct 30 19:45:04.485292 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.485272 4904 generic.go:356] "Generic (PLEG): container finished" podID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerID="b7aad3aa366c0ebc07dc22f91dabc248d34a7092c9159ee5465ba2d1ae1baef3" exitCode=60 Oct 30 19:45:04.485399 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.485326 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" event={"ID":"7ca5963c-22d7-4688-b367-1aa657eb37b4","Type":"ContainerDied","Data":"b7aad3aa366c0ebc07dc22f91dabc248d34a7092c9159ee5465ba2d1ae1baef3"} Oct 30 19:45:04.485594 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.485577 4904 scope.go:122] "RemoveContainer" containerID="b7aad3aa366c0ebc07dc22f91dabc248d34a7092c9159ee5465ba2d1ae1baef3" Oct 30 19:45:04.485838 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:04.485818 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl_odh-ai-gateway-infra(7ca5963c-22d7-4688-b367-1aa657eb37b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" Oct 30 19:45:04.492481 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.492461 4904 scope.go:122] "RemoveContainer" containerID="50cbc66691057e2354275f77a2a7604dc54e08e46342a155c55b977a610fe21d" Oct 30 19:45:04.505721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.505689 4904 scope.go:122] "RemoveContainer" containerID="809a89238b9c39ae9108818881b14cfb0ca85b94613e8e9b0bfb418239170b41" Oct 30 19:45:04.520753 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:04.520735 4904 scope.go:122] "RemoveContainer" containerID="7bbd662eba7cde8eb27a51c8729438a358775494b556489bbab76a326b14b04d" Oct 30 19:45:05.540853 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:05.540823 4904 scope.go:122] "RemoveContainer" containerID="e22ffdf8ca2cdf12035dd3cc13c435e298a27ab233cbfc5c112a1eb002589f3f" Oct 30 19:45:05.541749 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:05.541719 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p_odh-ai-gateway-infra(c1d48683-e2d8-4332-acb2-abd4133fff88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" Oct 30 19:45:05.558885 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:05.558857 4904 scope.go:122] "RemoveContainer" containerID="b7aad3aa366c0ebc07dc22f91dabc248d34a7092c9159ee5465ba2d1ae1baef3" Oct 30 19:45:05.559440 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:05.559413 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl_odh-ai-gateway-infra(7ca5963c-22d7-4688-b367-1aa657eb37b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" Oct 30 19:45:05.562622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:05.562603 4904 scope.go:122] "RemoveContainer" containerID="998f20c91fa81c457cf8d8cca30da06014f257847fe48a8642086f4db0a98f1e" Oct 30 19:45:05.563057 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:05.563035 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7_odh-ai-gateway-infra(74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" Oct 30 19:45:05.565600 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:05.565586 4904 scope.go:122] "RemoveContainer" containerID="a2df84e3e0f54ce168465b568f0c8e6be3bc0f22bde6f08351be6a1884e9f60a" Oct 30 19:45:05.565971 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:05.565953 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z_odh-ai-gateway-infra(05fe0b01-aaa5-4c59-9848-48c831f7dc4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" Oct 30 19:45:11.776212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:11.776176 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd"] Oct 30 19:45:11.776827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:11.776452 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" containerID="cri-o://c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84" gracePeriod=30 Oct 30 19:45:11.791508 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:11.791483 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx"] Oct 30 19:45:11.791696 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:11.791675 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" containerID="cri-o://47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c" gracePeriod=30 Oct 30 19:45:11.797950 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:11.797924 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw"] Oct 30 19:45:11.798228 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:11.798204 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" containerID="cri-o://14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86" gracePeriod=30 Oct 30 19:45:13.537062 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:13.537024 4904 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:45:13.537486 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:13.537089 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:45:17.118990 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.118959 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:45:17.182254 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.182221 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:45:17.247693 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.247663 4904 scope.go:122] "RemoveContainer" containerID="998f20c91fa81c457cf8d8cca30da06014f257847fe48a8642086f4db0a98f1e" Oct 30 19:45:17.248966 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.248939 4904 scope.go:122] "RemoveContainer" containerID="a2df84e3e0f54ce168465b568f0c8e6be3bc0f22bde6f08351be6a1884e9f60a" Oct 30 19:45:17.251249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.250938 4904 scope.go:122] "RemoveContainer" containerID="b7aad3aa366c0ebc07dc22f91dabc248d34a7092c9159ee5465ba2d1ae1baef3" Oct 30 19:45:17.256270 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.254862 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:45:17.257256 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.257225 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-socket\") pod \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " Oct 30 19:45:17.257512 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.257492 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-data\") pod \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " Oct 30 19:45:17.257587 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.257549 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istiod-ca-cert\") pod \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " Oct 30 19:45:17.257768 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.257592 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-credential-socket\") pod \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " Oct 30 19:45:17.257884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.257788 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-envoy\") pod \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " Oct 30 19:45:17.257884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.257818 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-podinfo\") pod \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " Oct 30 19:45:17.258067 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.258041 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-token\" (UniqueName: \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-token\") pod \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " Oct 30 19:45:17.261238 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.261184 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-certs\") pod \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " Oct 30 19:45:17.261594 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.261388 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-kube-api-access-4zck4\" (UniqueName: \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-kube-api-access-4zck4\") pod \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\" (UID: \"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef\") " Oct 30 19:45:17.261883 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.261856 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istiod-ca-cert" pod "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" (UID: "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:17.280131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.265468 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-socket" pod "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" (UID: "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.280131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.266081 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-data" pod "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" (UID: "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.280131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.275582 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-token" pod "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" (UID: "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:17.280131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.277456 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-certs" pod "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" (UID: "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.280131 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.278172 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-credential-socket" pod "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" (UID: "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.284406 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.284368 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-kube-api-access-4zck4" pod "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" (UID: "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef"). InnerVolumeSpecName "kube-api-access-4zck4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:17.284850 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.284830 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.299510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.299479 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-podinfo" pod "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" (UID: "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:17.301066 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.301038 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-envoy" pod "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" (UID: "7aa2d4f9-8b9e-4915-8753-62f185d5a7ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.385794 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.385762 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aad9d49-47d6-46b0-9d35-258081874d7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aad9d49-47d6-46b0-9d35-258081874d7c-istiod-ca-cert\") pod \"7aad9d49-47d6-46b0-9d35-258081874d7c\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " Oct 30 19:45:17.385913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.385814 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-data\") pod \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " Oct 30 19:45:17.385913 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.385851 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-podinfo\") pod \"7aad9d49-47d6-46b0-9d35-258081874d7c\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " Oct 30 19:45:17.386043 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.385930 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-credential-socket\") pod \"7aad9d49-47d6-46b0-9d35-258081874d7c\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " Oct 30 19:45:17.386043 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.385961 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-token\" (UniqueName: \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-token\") pod \"7aad9d49-47d6-46b0-9d35-258081874d7c\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " Oct 30 19:45:17.386043 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386001 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-kube-api-access-5fm9p\" (UniqueName: \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-kube-api-access-5fm9p\") pod \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " Oct 30 19:45:17.386043 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386033 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-credential-socket\") pod \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " Oct 30 19:45:17.386283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386087 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-socket\") pod \"7aad9d49-47d6-46b0-9d35-258081874d7c\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " Oct 30 19:45:17.386283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386144 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-kube-api-access-lln8c\" (UniqueName: \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-kube-api-access-lln8c\") pod \"7aad9d49-47d6-46b0-9d35-258081874d7c\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " Oct 30 19:45:17.386283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386148 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-data" pod "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" (UID: "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.386283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386193 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-data\") pod \"7aad9d49-47d6-46b0-9d35-258081874d7c\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " Oct 30 19:45:17.386283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386193 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aad9d49-47d6-46b0-9d35-258081874d7c-istiod-ca-cert" pod "7aad9d49-47d6-46b0-9d35-258081874d7c" (UID: "7aad9d49-47d6-46b0-9d35-258081874d7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:17.386283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386240 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-podinfo\") pod \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " Oct 30 19:45:17.386561 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386299 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-token\" (UniqueName: \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-token\") pod \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " Oct 30 19:45:17.386561 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386326 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-socket\") pod \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " Oct 30 19:45:17.386561 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386386 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-credential-socket" pod "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" (UID: "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.386561 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386452 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istiod-ca-cert\") pod \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " Oct 30 19:45:17.386561 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386483 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-certs\") pod \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " Oct 30 19:45:17.386742 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386566 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-certs\") pod \"7aad9d49-47d6-46b0-9d35-258081874d7c\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " Oct 30 19:45:17.386742 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386574 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-credential-socket" pod "7aad9d49-47d6-46b0-9d35-258081874d7c" (UID: "7aad9d49-47d6-46b0-9d35-258081874d7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.386742 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386638 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-envoy\") pod \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\" (UID: \"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65\") " Oct 30 19:45:17.386742 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386675 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-envoy\") pod \"7aad9d49-47d6-46b0-9d35-258081874d7c\" (UID: \"7aad9d49-47d6-46b0-9d35-258081874d7c\") " Oct 30 19:45:17.386903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.386866 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-data" pod "7aad9d49-47d6-46b0-9d35-258081874d7c" (UID: "7aad9d49-47d6-46b0-9d35-258081874d7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.387667 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.387473 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-certs" pod "7aad9d49-47d6-46b0-9d35-258081874d7c" (UID: "7aad9d49-47d6-46b0-9d35-258081874d7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.387667 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.387600 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istiod-ca-cert" pod "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" (UID: "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:17.387926 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.387886 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-certs" pod "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" (UID: "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.387926 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.387899 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-socket" pod "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" (UID: "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.388740 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388716 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388745 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388762 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388775 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aad9d49-47d6-46b0-9d35-258081874d7c-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388788 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388800 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388813 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zck4\" (UniqueName: \"kubernetes.io/projected/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-kube-api-access-4zck4\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388826 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388838 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388851 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388864 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388876 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388888 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.388902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.388901 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.390092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.389879 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-podinfo" pod "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" (UID: "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:17.390092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.389892 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-envoy" pod "7aad9d49-47d6-46b0-9d35-258081874d7c" (UID: "7aad9d49-47d6-46b0-9d35-258081874d7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.390092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.389912 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-token" pod "7aad9d49-47d6-46b0-9d35-258081874d7c" (UID: "7aad9d49-47d6-46b0-9d35-258081874d7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:17.390092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.390012 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-kube-api-access-lln8c" pod "7aad9d49-47d6-46b0-9d35-258081874d7c" (UID: "7aad9d49-47d6-46b0-9d35-258081874d7c"). InnerVolumeSpecName "kube-api-access-lln8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:17.390331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.390191 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-kube-api-access-5fm9p" pod "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" (UID: "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65"). InnerVolumeSpecName "kube-api-access-5fm9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:17.390331 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.390279 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-socket" pod "7aad9d49-47d6-46b0-9d35-258081874d7c" (UID: "7aad9d49-47d6-46b0-9d35-258081874d7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.390877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.390856 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-envoy" pod "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" (UID: "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.390957 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.390944 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-token" pod "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" (UID: "9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:17.391843 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.391800 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-podinfo" pod "7aad9d49-47d6-46b0-9d35-258081874d7c" (UID: "7aad9d49-47d6-46b0-9d35-258081874d7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:17.490994 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.490966 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.490994 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.490992 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.490994 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.491002 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.491305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.491011 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.491305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.491019 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.491305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.491026 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.491305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.491034 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.491305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.491042 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.491305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.491054 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fm9p\" (UniqueName: \"kubernetes.io/projected/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-kube-api-access-5fm9p\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.491305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.491068 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aad9d49-47d6-46b0-9d35-258081874d7c-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.491305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.491077 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lln8c\" (UniqueName: \"kubernetes.io/projected/7aad9d49-47d6-46b0-9d35-258081874d7c-kube-api-access-lln8c\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.491305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.491085 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.674933 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.674898 4904 generic.go:356] "Generic (PLEG): container finished" podID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerID="47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c" exitCode=0 Oct 30 19:45:17.675157 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.674956 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" Oct 30 19:45:17.675157 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.674973 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" event={"ID":"7aad9d49-47d6-46b0-9d35-258081874d7c","Type":"ContainerDied","Data":"47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c"} Oct 30 19:45:17.675157 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.675029 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx" event={"ID":"7aad9d49-47d6-46b0-9d35-258081874d7c","Type":"ContainerDied","Data":"68cf108198b62268019ee8324731250959ac933641e26de4cba7f246d9d337b4"} Oct 30 19:45:17.675157 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.675054 4904 scope.go:122] "RemoveContainer" containerID="47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c" Oct 30 19:45:17.676873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.676777 4904 generic.go:356] "Generic (PLEG): container finished" podID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerID="c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84" exitCode=0 Oct 30 19:45:17.676873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.676824 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" event={"ID":"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef","Type":"ContainerDied","Data":"c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84"} Oct 30 19:45:17.677022 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.676844 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" Oct 30 19:45:17.677022 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.676914 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd" event={"ID":"7aa2d4f9-8b9e-4915-8753-62f185d5a7ef","Type":"ContainerDied","Data":"cbbdcb1c57c92c82de98885bd26db13d157cfa16f8dcfb1e2d4a85fcda888837"} Oct 30 19:45:17.679872 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.679846 4904 generic.go:356] "Generic (PLEG): container finished" podID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerID="14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86" exitCode=0 Oct 30 19:45:17.679982 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.679953 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" event={"ID":"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65","Type":"ContainerDied","Data":"14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86"} Oct 30 19:45:17.680023 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.679992 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" event={"ID":"9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65","Type":"ContainerDied","Data":"af21396814e72ffcd45fbf24b7ea391c5ede08cfdc83a5574101676da61928c3"} Oct 30 19:45:17.680092 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.680079 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw" Oct 30 19:45:17.691213 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.691185 4904 scope.go:122] "RemoveContainer" containerID="47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c" Oct 30 19:45:17.691440 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:17.691423 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c\": container with ID starting with 47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c not found: ID does not exist" containerID="47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c" Oct 30 19:45:17.691500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.691449 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c"} err="failed to get container status \"47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c\": rpc error: code = NotFound desc = could not find container \"47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c\": container with ID starting with 47a1ebaf717e488c765dd7bb5674356c7d95aa8ba7ef5e6b1f9fe97cfd8bf45c not found: ID does not exist" Oct 30 19:45:17.691500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.691467 4904 scope.go:122] "RemoveContainer" containerID="c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84" Oct 30 19:45:17.701895 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.701863 4904 scope.go:122] "RemoveContainer" containerID="c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84" Oct 30 19:45:17.702174 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:17.702155 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84\": container with ID starting with c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84 not found: ID does not exist" containerID="c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84" Oct 30 19:45:17.702265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.702185 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84"} err="failed to get container status \"c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84\": rpc error: code = NotFound desc = could not find container \"c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84\": container with ID starting with c444fe2c619a495f60f70b108c33ee0cab9c08939f841bb1286f2f058d5aab84 not found: ID does not exist" Oct 30 19:45:17.702265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.702207 4904 scope.go:122] "RemoveContainer" containerID="14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86" Oct 30 19:45:17.712979 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.712960 4904 scope.go:122] "RemoveContainer" containerID="14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86" Oct 30 19:45:17.713314 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:17.713295 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86\": container with ID starting with 14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86 not found: ID does not exist" containerID="14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86" Oct 30 19:45:17.713388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.713322 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86"} err="failed to get container status \"14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86\": rpc error: code = NotFound desc = could not find container \"14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86\": container with ID starting with 14fdb9e2f0182291814d8c17aaa4ca80014923970836d15bf1ff0a98a7deaf86 not found: ID does not exist" Oct 30 19:45:17.754379 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.754356 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd"] Oct 30 19:45:17.767907 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.767885 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7rzpd"] Oct 30 19:45:17.805725 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.805698 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx"] Oct 30 19:45:17.809180 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.809161 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-mkmnx"] Oct 30 19:45:17.849077 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.849004 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw"] Oct 30 19:45:17.854950 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:17.854927 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-v79sw"] Oct 30 19:45:18.215568 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.215539 4904 scope.go:122] "RemoveContainer" containerID="e22ffdf8ca2cdf12035dd3cc13c435e298a27ab233cbfc5c112a1eb002589f3f" Oct 30 19:45:18.690449 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.690419 4904 generic.go:356] "Generic (PLEG): container finished" podID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerID="58ac00e6857b15abae8490da7197b029e7988bb3d586873b642cc031e61893be" exitCode=60 Oct 30 19:45:18.690613 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.690498 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" event={"ID":"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1","Type":"ContainerDied","Data":"58ac00e6857b15abae8490da7197b029e7988bb3d586873b642cc031e61893be"} Oct 30 19:45:18.690613 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.690562 4904 scope.go:122] "RemoveContainer" containerID="998f20c91fa81c457cf8d8cca30da06014f257847fe48a8642086f4db0a98f1e" Oct 30 19:45:18.691061 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.691032 4904 scope.go:122] "RemoveContainer" containerID="58ac00e6857b15abae8490da7197b029e7988bb3d586873b642cc031e61893be" Oct 30 19:45:18.691458 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:18.691436 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7_odh-ai-gateway-infra(74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" Oct 30 19:45:18.694648 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.694627 4904 generic.go:356] "Generic (PLEG): container finished" podID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerID="2bc6b7984f57626bc09c37e24a309faa772993afdf5112cede385f83eb4b9f4e" exitCode=60 Oct 30 19:45:18.694770 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.694694 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" event={"ID":"05fe0b01-aaa5-4c59-9848-48c831f7dc4d","Type":"ContainerDied","Data":"2bc6b7984f57626bc09c37e24a309faa772993afdf5112cede385f83eb4b9f4e"} Oct 30 19:45:18.695093 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.695076 4904 scope.go:122] "RemoveContainer" containerID="2bc6b7984f57626bc09c37e24a309faa772993afdf5112cede385f83eb4b9f4e" Oct 30 19:45:18.695441 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:18.695418 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z_odh-ai-gateway-infra(05fe0b01-aaa5-4c59-9848-48c831f7dc4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" Oct 30 19:45:18.696926 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.696901 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" event={"ID":"c1d48683-e2d8-4332-acb2-abd4133fff88","Type":"ContainerStarted","Data":"7becbdd0b2f74d9529e3f3227e84a06da96ffa9381d429b73e9b3bcd87f822a9"} Oct 30 19:45:18.698892 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.698871 4904 generic.go:356] "Generic (PLEG): container finished" podID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerID="146a23069190673ca8c98484fd23cabfa16ae9c10433e40b156b5ce9f6f6fa1c" exitCode=60 Oct 30 19:45:18.699008 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.698911 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" event={"ID":"7ca5963c-22d7-4688-b367-1aa657eb37b4","Type":"ContainerDied","Data":"146a23069190673ca8c98484fd23cabfa16ae9c10433e40b156b5ce9f6f6fa1c"} Oct 30 19:45:18.699280 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.699265 4904 scope.go:122] "RemoveContainer" containerID="146a23069190673ca8c98484fd23cabfa16ae9c10433e40b156b5ce9f6f6fa1c" Oct 30 19:45:18.699562 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:18.699544 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl_odh-ai-gateway-infra(7ca5963c-22d7-4688-b367-1aa657eb37b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" Oct 30 19:45:18.705817 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.705802 4904 scope.go:122] "RemoveContainer" containerID="a2df84e3e0f54ce168465b568f0c8e6be3bc0f22bde6f08351be6a1884e9f60a" Oct 30 19:45:18.724832 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:18.724755 4904 scope.go:122] "RemoveContainer" containerID="b7aad3aa366c0ebc07dc22f91dabc248d34a7092c9159ee5465ba2d1ae1baef3" Oct 30 19:45:19.226366 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.226292 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" path="/var/lib/kubelet/pods/7aa2d4f9-8b9e-4915-8753-62f185d5a7ef/volumes" Oct 30 19:45:19.226793 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.226778 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" path="/var/lib/kubelet/pods/7aad9d49-47d6-46b0-9d35-258081874d7c/volumes" Oct 30 19:45:19.227378 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.227359 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" path="/var/lib/kubelet/pods/9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65/volumes" Oct 30 19:45:19.710167 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.710118 4904 generic.go:356] "Generic (PLEG): container finished" podID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerID="7becbdd0b2f74d9529e3f3227e84a06da96ffa9381d429b73e9b3bcd87f822a9" exitCode=60 Oct 30 19:45:19.710365 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.710186 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" event={"ID":"c1d48683-e2d8-4332-acb2-abd4133fff88","Type":"ContainerDied","Data":"7becbdd0b2f74d9529e3f3227e84a06da96ffa9381d429b73e9b3bcd87f822a9"} Oct 30 19:45:19.710365 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.710253 4904 scope.go:122] "RemoveContainer" containerID="e22ffdf8ca2cdf12035dd3cc13c435e298a27ab233cbfc5c112a1eb002589f3f" Oct 30 19:45:19.710691 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.710663 4904 scope.go:122] "RemoveContainer" containerID="7becbdd0b2f74d9529e3f3227e84a06da96ffa9381d429b73e9b3bcd87f822a9" Oct 30 19:45:19.711092 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:19.711054 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p_odh-ai-gateway-infra(c1d48683-e2d8-4332-acb2-abd4133fff88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" Oct 30 19:45:19.780792 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.780766 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7"] Oct 30 19:45:19.813657 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.813626 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl"] Oct 30 19:45:19.868252 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.847901 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p"] Oct 30 19:45:19.904704 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:19.904667 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z"] Oct 30 19:45:20.002442 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.002412 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" Oct 30 19:45:20.048928 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.048896 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-maas-api-service-ca\") pod \"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1\" (UID: \"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1\") " Oct 30 19:45:20.051896 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.051867 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-kube-api-access-kkb99\" (UniqueName: \"kubernetes.io/projected/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-kube-api-access-kkb99\") pod \"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1\" (UID: \"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1\") " Oct 30 19:45:20.054016 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.053992 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-maas-api-service-ca" pod "74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" (UID: "74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.069545 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.069517 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-kube-api-access-kkb99" pod "74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" (UID: "74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1"). InnerVolumeSpecName "kube-api-access-kkb99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.112945 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.112920 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" Oct 30 19:45:20.154211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.154182 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca5963c-22d7-4688-b367-1aa657eb37b4-kube-api-access-z5pvt\" (UniqueName: \"kubernetes.io/projected/7ca5963c-22d7-4688-b367-1aa657eb37b4-kube-api-access-z5pvt\") pod \"7ca5963c-22d7-4688-b367-1aa657eb37b4\" (UID: \"7ca5963c-22d7-4688-b367-1aa657eb37b4\") " Oct 30 19:45:20.154342 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.154233 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca5963c-22d7-4688-b367-1aa657eb37b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca5963c-22d7-4688-b367-1aa657eb37b4-maas-api-service-ca\") pod \"7ca5963c-22d7-4688-b367-1aa657eb37b4\" (UID: \"7ca5963c-22d7-4688-b367-1aa657eb37b4\") " Oct 30 19:45:20.154607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.154571 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca5963c-22d7-4688-b367-1aa657eb37b4-maas-api-service-ca" pod "7ca5963c-22d7-4688-b367-1aa657eb37b4" (UID: "7ca5963c-22d7-4688-b367-1aa657eb37b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.154737 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.154721 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" Oct 30 19:45:20.154825 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.154804 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-maas-api-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.154825 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.154824 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca5963c-22d7-4688-b367-1aa657eb37b4-maas-api-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.154962 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.154833 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkb99\" (UniqueName: \"kubernetes.io/projected/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1-kube-api-access-kkb99\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.156893 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.156872 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca5963c-22d7-4688-b367-1aa657eb37b4-kube-api-access-z5pvt" pod "7ca5963c-22d7-4688-b367-1aa657eb37b4" (UID: "7ca5963c-22d7-4688-b367-1aa657eb37b4"). InnerVolumeSpecName "kube-api-access-z5pvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.256768 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.256683 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-kube-api-access-ngfhr\" (UniqueName: \"kubernetes.io/projected/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-kube-api-access-ngfhr\") pod \"05fe0b01-aaa5-4c59-9848-48c831f7dc4d\" (UID: \"05fe0b01-aaa5-4c59-9848-48c831f7dc4d\") " Oct 30 19:45:20.257271 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.256923 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-maas-api-service-ca\") pod \"05fe0b01-aaa5-4c59-9848-48c831f7dc4d\" (UID: \"05fe0b01-aaa5-4c59-9848-48c831f7dc4d\") " Oct 30 19:45:20.257739 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.257714 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5pvt\" (UniqueName: \"kubernetes.io/projected/7ca5963c-22d7-4688-b367-1aa657eb37b4-kube-api-access-z5pvt\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.258185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.258160 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-maas-api-service-ca" pod "05fe0b01-aaa5-4c59-9848-48c831f7dc4d" (UID: "05fe0b01-aaa5-4c59-9848-48c831f7dc4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.261235 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.261209 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-kube-api-access-ngfhr" pod "05fe0b01-aaa5-4c59-9848-48c831f7dc4d" (UID: "05fe0b01-aaa5-4c59-9848-48c831f7dc4d"). InnerVolumeSpecName "kube-api-access-ngfhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.359162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.359135 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngfhr\" (UniqueName: \"kubernetes.io/projected/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-kube-api-access-ngfhr\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.359283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.359166 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05fe0b01-aaa5-4c59-9848-48c831f7dc4d-maas-api-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.728503 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.728475 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" Oct 30 19:45:20.728662 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.728532 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl" event={"ID":"7ca5963c-22d7-4688-b367-1aa657eb37b4","Type":"ContainerDied","Data":"a9e5805e278f1959775f8ff01cbffc2f33366926e806d94e8f338cbf4b031a42"} Oct 30 19:45:20.728662 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.728657 4904 scope.go:122] "RemoveContainer" containerID="146a23069190673ca8c98484fd23cabfa16ae9c10433e40b156b5ce9f6f6fa1c" Oct 30 19:45:20.733139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.730413 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" Oct 30 19:45:20.733139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.730441 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7" event={"ID":"74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1","Type":"ContainerDied","Data":"882081dd9ab4b27b853ca33e766df051247e7f9b2c2ad309695e712eda737d7e"} Oct 30 19:45:20.736191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.736143 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" event={"ID":"05fe0b01-aaa5-4c59-9848-48c831f7dc4d","Type":"ContainerDied","Data":"ab74474393ca16a984856e2b6e5db4c8f3b7d2c4d15347daebb5543fbf547ca2"} Oct 30 19:45:20.736594 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.736514 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z" Oct 30 19:45:20.751882 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.751856 4904 scope.go:122] "RemoveContainer" containerID="58ac00e6857b15abae8490da7197b029e7988bb3d586873b642cc031e61893be" Oct 30 19:45:20.768602 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.768581 4904 scope.go:122] "RemoveContainer" containerID="2bc6b7984f57626bc09c37e24a309faa772993afdf5112cede385f83eb4b9f4e" Oct 30 19:45:20.830827 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.830797 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z"] Oct 30 19:45:20.837024 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.837000 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3735b2-29889825-hm69z"] Oct 30 19:45:20.871556 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.871505 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl"] Oct 30 19:45:20.900166 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.900140 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3add95-29889825-x48rl"] Oct 30 19:45:20.911493 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.911467 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" Oct 30 19:45:20.965738 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.965710 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7"] Oct 30 19:45:20.966222 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.966201 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d48683-e2d8-4332-acb2-abd4133fff88-kube-api-access-hlnqr\" (UniqueName: \"kubernetes.io/projected/c1d48683-e2d8-4332-acb2-abd4133fff88-kube-api-access-hlnqr\") pod \"c1d48683-e2d8-4332-acb2-abd4133fff88\" (UID: \"c1d48683-e2d8-4332-acb2-abd4133fff88\") " Oct 30 19:45:20.966532 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.966499 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d48683-e2d8-4332-acb2-abd4133fff88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d48683-e2d8-4332-acb2-abd4133fff88-maas-api-service-ca\") pod \"c1d48683-e2d8-4332-acb2-abd4133fff88\" (UID: \"c1d48683-e2d8-4332-acb2-abd4133fff88\") " Oct 30 19:45:20.967067 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.967041 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d48683-e2d8-4332-acb2-abd4133fff88-maas-api-service-ca" pod "c1d48683-e2d8-4332-acb2-abd4133fff88" (UID: "c1d48683-e2d8-4332-acb2-abd4133fff88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.968565 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.968531 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d48683-e2d8-4332-acb2-abd4133fff88-kube-api-access-hlnqr" pod "c1d48683-e2d8-4332-acb2-abd4133fff88" (UID: "c1d48683-e2d8-4332-acb2-abd4133fff88"). InnerVolumeSpecName "kube-api-access-hlnqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.975494 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:20.975472 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-086f2a-29889825-x89c7"] Oct 30 19:45:21.068384 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:21.068340 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d48683-e2d8-4332-acb2-abd4133fff88-maas-api-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.068384 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:21.068387 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlnqr\" (UniqueName: \"kubernetes.io/projected/c1d48683-e2d8-4332-acb2-abd4133fff88-kube-api-access-hlnqr\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.225411 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:21.225380 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" path="/var/lib/kubelet/pods/05fe0b01-aaa5-4c59-9848-48c831f7dc4d/volumes" Oct 30 19:45:21.226229 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:21.226203 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" path="/var/lib/kubelet/pods/74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1/volumes" Oct 30 19:45:21.226748 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:21.226724 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" path="/var/lib/kubelet/pods/7ca5963c-22d7-4688-b367-1aa657eb37b4/volumes" Oct 30 19:45:21.760677 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:21.760646 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" Oct 30 19:45:21.761200 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:21.760646 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p" event={"ID":"c1d48683-e2d8-4332-acb2-abd4133fff88","Type":"ContainerDied","Data":"fc3d94da4ee1069db6b7f0048b06161bb3ff39755ec6c2d7083647ca364b90cb"} Oct 30 19:45:21.761288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:21.761269 4904 scope.go:122] "RemoveContainer" containerID="7becbdd0b2f74d9529e3f3227e84a06da96ffa9381d429b73e9b3bcd87f822a9" Oct 30 19:45:21.839727 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:21.839699 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p"] Oct 30 19:45:21.845858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:21.845832 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b4c34-29889825-znx8p"] Oct 30 19:45:23.234258 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:23.234225 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" path="/var/lib/kubelet/pods/c1d48683-e2d8-4332-acb2-abd4133fff88/volumes" Oct 30 19:45:24.510185 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:24.510151 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:45:32.563128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:32.562654 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw"] Oct 30 19:45:39.553901 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:39.553871 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:45:39.560324 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:39.560300 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:45:39.572855 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:39.572834 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:45:39.578737 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:39.578703 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:45:47.601745 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.601704 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" containerID="cri-o://b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599" gracePeriod=45 Oct 30 19:45:47.880793 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.880769 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:45:47.934605 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.934576 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-dshm\") pod \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " Oct 30 19:45:47.934738 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.934658 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-home\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-home\") pod \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " Oct 30 19:45:47.934738 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.934695 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tmp-dir\") pod \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " Oct 30 19:45:47.934833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.934774 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tls-certs\") pod \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " Oct 30 19:45:47.934894 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.934833 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-kube-api-access-dcnl4\" (UniqueName: \"kubernetes.io/projected/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-kube-api-access-dcnl4\") pod \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " Oct 30 19:45:47.934894 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.934869 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-model-cache\") pod \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\" (UID: \"0c3b2f6a-9984-4f4f-9874-24a3bf955a27\") " Oct 30 19:45:47.935025 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.934998 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-home" pod "0c3b2f6a-9984-4f4f-9874-24a3bf955a27" (UID: "0c3b2f6a-9984-4f4f-9874-24a3bf955a27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:47.935283 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.935258 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-model-cache" pod "0c3b2f6a-9984-4f4f-9874-24a3bf955a27" (UID: "0c3b2f6a-9984-4f4f-9874-24a3bf955a27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:47.935411 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.935371 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tmp-dir" pod "0c3b2f6a-9984-4f4f-9874-24a3bf955a27" (UID: "0c3b2f6a-9984-4f4f-9874-24a3bf955a27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:47.935599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.935546 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:47.935699 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.935607 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:47.935699 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.935621 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:47.938135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.938073 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tls-certs" pod "0c3b2f6a-9984-4f4f-9874-24a3bf955a27" (UID: "0c3b2f6a-9984-4f4f-9874-24a3bf955a27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:47.938226 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.938141 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-kube-api-access-dcnl4" pod "0c3b2f6a-9984-4f4f-9874-24a3bf955a27" (UID: "0c3b2f6a-9984-4f4f-9874-24a3bf955a27"). InnerVolumeSpecName "kube-api-access-dcnl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:47.938226 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:47.938202 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-dshm" pod "0c3b2f6a-9984-4f4f-9874-24a3bf955a27" (UID: "0c3b2f6a-9984-4f4f-9874-24a3bf955a27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:48.037055 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.037017 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:48.037055 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.037052 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcnl4\" (UniqueName: \"kubernetes.io/projected/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-kube-api-access-dcnl4\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:48.037275 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.037069 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3b2f6a-9984-4f4f-9874-24a3bf955a27-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:45:48.087376 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.087325 4904 generic.go:356] "Generic (PLEG): container finished" podID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerID="b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599" exitCode=0 Oct 30 19:45:48.087376 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.087372 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" Oct 30 19:45:48.087615 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.087529 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" event={"ID":"0c3b2f6a-9984-4f4f-9874-24a3bf955a27","Type":"ContainerDied","Data":"b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599"} Oct 30 19:45:48.087615 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.087560 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw" event={"ID":"0c3b2f6a-9984-4f4f-9874-24a3bf955a27","Type":"ContainerDied","Data":"a64f8f74a8537a8a41c23a1eb5e038452d8830aad17c0df59cd38fe345c874e0"} Oct 30 19:45:48.087615 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.087578 4904 scope.go:122] "RemoveContainer" containerID="b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599" Oct 30 19:45:48.099266 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.099244 4904 scope.go:122] "RemoveContainer" containerID="b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599" Oct 30 19:45:48.099573 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:48.099553 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599\": container with ID starting with b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599 not found: ID does not exist" containerID="b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599" Oct 30 19:45:48.099692 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.099583 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599"} err="failed to get container status \"b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599\": rpc error: code = NotFound desc = could not find container \"b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599\": container with ID starting with b21b3bb10528d979d3c90b3d51557fa5227ead48de3fbe495ac4a364e6332599 not found: ID does not exist" Oct 30 19:45:48.168362 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.168282 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw"] Oct 30 19:45:48.176129 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:48.176080 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-w2-3735b2-kserve-6f6f588d9c-764rw"] Oct 30 19:45:49.225038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:49.225004 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" path="/var/lib/kubelet/pods/0c3b2f6a-9984-4f4f-9874-24a3bf955a27/volumes" Oct 30 19:45:54.510780 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:45:54.510746 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:45:58.096079 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:58.096044 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj"] Oct 30 19:45:58.113191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:45:58.113161 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f"] Oct 30 19:46:13.121393 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.121350 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" containerID="cri-o://375dc63220f78823ba5d487c692898ecdbc0f78deecbe9159b9ded5cff160a0c" gracePeriod=45 Oct 30 19:46:13.143597 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.143562 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" containerID="cri-o://fc563fea9cb8854c3b04b03b49025d3b65468da06402a3a9dc2c2981a5424f56" gracePeriod=45 Oct 30 19:46:13.363094 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.363054 4904 generic.go:356] "Generic (PLEG): container finished" podID="c37ac3f1-7271-41db-8d39-9194a2820225" containerID="fc563fea9cb8854c3b04b03b49025d3b65468da06402a3a9dc2c2981a5424f56" exitCode=0 Oct 30 19:46:13.363306 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.363211 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" event={"ID":"c37ac3f1-7271-41db-8d39-9194a2820225","Type":"ContainerDied","Data":"fc563fea9cb8854c3b04b03b49025d3b65468da06402a3a9dc2c2981a5424f56"} Oct 30 19:46:13.365088 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.364886 4904 generic.go:356] "Generic (PLEG): container finished" podID="87aab154-3c67-4749-a67d-365be3e3437d" containerID="375dc63220f78823ba5d487c692898ecdbc0f78deecbe9159b9ded5cff160a0c" exitCode=0 Oct 30 19:46:13.365088 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.365056 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" event={"ID":"87aab154-3c67-4749-a67d-365be3e3437d","Type":"ContainerDied","Data":"375dc63220f78823ba5d487c692898ecdbc0f78deecbe9159b9ded5cff160a0c"} Oct 30 19:46:13.480971 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.480933 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:46:13.486457 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.486435 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:46:13.500389 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500359 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37ac3f1-7271-41db-8d39-9194a2820225-kube-api-access-vqggl\" (UniqueName: \"kubernetes.io/projected/c37ac3f1-7271-41db-8d39-9194a2820225-kube-api-access-vqggl\") pod \"c37ac3f1-7271-41db-8d39-9194a2820225\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " Oct 30 19:46:13.500528 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500401 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c37ac3f1-7271-41db-8d39-9194a2820225-tls-certs\" (UniqueName: \"kubernetes.io/secret/c37ac3f1-7271-41db-8d39-9194a2820225-tls-certs\") pod \"c37ac3f1-7271-41db-8d39-9194a2820225\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " Oct 30 19:46:13.500528 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500435 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-dshm\") pod \"87aab154-3c67-4749-a67d-365be3e3437d\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " Oct 30 19:46:13.500528 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500476 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-tmp-dir\") pod \"87aab154-3c67-4749-a67d-365be3e3437d\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " Oct 30 19:46:13.500528 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500511 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87aab154-3c67-4749-a67d-365be3e3437d-tls-certs\" (UniqueName: \"kubernetes.io/secret/87aab154-3c67-4749-a67d-365be3e3437d-tls-certs\") pod \"87aab154-3c67-4749-a67d-365be3e3437d\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " Oct 30 19:46:13.500773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500559 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-model-cache\") pod \"c37ac3f1-7271-41db-8d39-9194a2820225\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " Oct 30 19:46:13.500773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500613 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-model-cache\") pod \"87aab154-3c67-4749-a67d-365be3e3437d\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " Oct 30 19:46:13.500773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500698 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-tmp-dir\") pod \"c37ac3f1-7271-41db-8d39-9194a2820225\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " Oct 30 19:46:13.500773 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500738 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87aab154-3c67-4749-a67d-365be3e3437d-kube-api-access-xk4cj\" (UniqueName: \"kubernetes.io/projected/87aab154-3c67-4749-a67d-365be3e3437d-kube-api-access-xk4cj\") pod \"87aab154-3c67-4749-a67d-365be3e3437d\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " Oct 30 19:46:13.501132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500831 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-home\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-home\") pod \"c37ac3f1-7271-41db-8d39-9194a2820225\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " Oct 30 19:46:13.501132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500939 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-home\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-home\") pod \"87aab154-3c67-4749-a67d-365be3e3437d\" (UID: \"87aab154-3c67-4749-a67d-365be3e3437d\") " Oct 30 19:46:13.501132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.500981 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-dshm\") pod \"c37ac3f1-7271-41db-8d39-9194a2820225\" (UID: \"c37ac3f1-7271-41db-8d39-9194a2820225\") " Oct 30 19:46:13.501909 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.501666 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-home" pod "c37ac3f1-7271-41db-8d39-9194a2820225" (UID: "c37ac3f1-7271-41db-8d39-9194a2820225"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:13.501909 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.501711 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-model-cache" pod "c37ac3f1-7271-41db-8d39-9194a2820225" (UID: "c37ac3f1-7271-41db-8d39-9194a2820225"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:13.501909 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.501726 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-model-cache" pod "87aab154-3c67-4749-a67d-365be3e3437d" (UID: "87aab154-3c67-4749-a67d-365be3e3437d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:13.501909 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.501749 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-tmp-dir" pod "c37ac3f1-7271-41db-8d39-9194a2820225" (UID: "c37ac3f1-7271-41db-8d39-9194a2820225"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:13.502133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.501955 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-home" pod "87aab154-3c67-4749-a67d-365be3e3437d" (UID: "87aab154-3c67-4749-a67d-365be3e3437d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:13.502133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.501969 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-tmp-dir" pod "87aab154-3c67-4749-a67d-365be3e3437d" (UID: "87aab154-3c67-4749-a67d-365be3e3437d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:13.505559 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.505534 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-dshm" pod "c37ac3f1-7271-41db-8d39-9194a2820225" (UID: "c37ac3f1-7271-41db-8d39-9194a2820225"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:13.506720 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.506697 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87aab154-3c67-4749-a67d-365be3e3437d-kube-api-access-xk4cj" pod "87aab154-3c67-4749-a67d-365be3e3437d" (UID: "87aab154-3c67-4749-a67d-365be3e3437d"). InnerVolumeSpecName "kube-api-access-xk4cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:13.509448 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.509383 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87aab154-3c67-4749-a67d-365be3e3437d-tls-certs" pod "87aab154-3c67-4749-a67d-365be3e3437d" (UID: "87aab154-3c67-4749-a67d-365be3e3437d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:13.509448 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.509396 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37ac3f1-7271-41db-8d39-9194a2820225-tls-certs" pod "c37ac3f1-7271-41db-8d39-9194a2820225" (UID: "c37ac3f1-7271-41db-8d39-9194a2820225"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:13.509612 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.509474 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37ac3f1-7271-41db-8d39-9194a2820225-kube-api-access-vqggl" pod "c37ac3f1-7271-41db-8d39-9194a2820225" (UID: "c37ac3f1-7271-41db-8d39-9194a2820225"). InnerVolumeSpecName "kube-api-access-vqggl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:13.511151 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.511123 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-dshm" pod "87aab154-3c67-4749-a67d-365be3e3437d" (UID: "87aab154-3c67-4749-a67d-365be3e3437d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:13.602676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602643 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.602676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602672 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk4cj\" (UniqueName: \"kubernetes.io/projected/87aab154-3c67-4749-a67d-365be3e3437d-kube-api-access-xk4cj\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.602676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602681 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.602676 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602689 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.603031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602697 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.603031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602707 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqggl\" (UniqueName: \"kubernetes.io/projected/c37ac3f1-7271-41db-8d39-9194a2820225-kube-api-access-vqggl\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.603031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602716 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c37ac3f1-7271-41db-8d39-9194a2820225-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.603031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602726 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.603031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602734 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.603031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602741 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87aab154-3c67-4749-a67d-365be3e3437d-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.603031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602748 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c37ac3f1-7271-41db-8d39-9194a2820225-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:13.603031 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:13.602757 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87aab154-3c67-4749-a67d-365be3e3437d-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:14.373319 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:14.373284 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" Oct 30 19:46:14.373804 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:14.373315 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f" event={"ID":"c37ac3f1-7271-41db-8d39-9194a2820225","Type":"ContainerDied","Data":"db582b4f18d1a252b2564ed3a302929375039270ed3cbbcfee38163f9d6b75bd"} Oct 30 19:46:14.373804 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:14.373377 4904 scope.go:122] "RemoveContainer" containerID="fc563fea9cb8854c3b04b03b49025d3b65468da06402a3a9dc2c2981a5424f56" Oct 30 19:46:14.374767 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:14.374744 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" event={"ID":"87aab154-3c67-4749-a67d-365be3e3437d","Type":"ContainerDied","Data":"118094aeb52feef6a8cf1ba811fbac415d918b6a84c8e6c86435422e056fdbd6"} Oct 30 19:46:14.374767 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:14.374757 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj" Oct 30 19:46:14.386223 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:14.386205 4904 scope.go:122] "RemoveContainer" containerID="375dc63220f78823ba5d487c692898ecdbc0f78deecbe9159b9ded5cff160a0c" Oct 30 19:46:14.430981 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:14.430961 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f"] Oct 30 19:46:14.446965 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:14.446937 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/simulator-w6-ef9a5a-kserve-5d85f478b8-nsl6f"] Oct 30 19:46:14.473849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:14.473828 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj"] Oct 30 19:46:14.477118 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:14.477088 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/premium-w6-ef9a5a-kserve-59bd8957fc-5rfvj"] Oct 30 19:46:15.222589 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:15.222561 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87aab154-3c67-4749-a67d-365be3e3437d" path="/var/lib/kubelet/pods/87aab154-3c67-4749-a67d-365be3e3437d/volumes" Oct 30 19:46:15.223176 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:15.223157 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" path="/var/lib/kubelet/pods/c37ac3f1-7271-41db-8d39-9194a2820225/volumes" Oct 30 19:46:17.389188 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:17.389147 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m"] Oct 30 19:46:17.389656 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:17.389403 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" containerID="cri-o://405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a" gracePeriod=30 Oct 30 19:46:21.454983 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:21.454831 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg"] Oct 30 19:46:21.456371 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:21.456336 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" containerID="cri-o://f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7" gracePeriod=30 Oct 30 19:46:22.683763 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.683739 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:46:22.854223 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.854186 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-token\" (UniqueName: \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-token\") pod \"7cba5790-7577-462a-aec0-759ff1d6bb40\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " Oct 30 19:46:22.854406 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.854279 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-kube-api-access-f9rzc\" (UniqueName: \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-kube-api-access-f9rzc\") pod \"7cba5790-7577-462a-aec0-759ff1d6bb40\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " Oct 30 19:46:22.854406 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.854372 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cba5790-7577-462a-aec0-759ff1d6bb40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cba5790-7577-462a-aec0-759ff1d6bb40-istiod-ca-cert\") pod \"7cba5790-7577-462a-aec0-759ff1d6bb40\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " Oct 30 19:46:22.854525 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.854405 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-credential-socket\") pod \"7cba5790-7577-462a-aec0-759ff1d6bb40\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " Oct 30 19:46:22.854525 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.854447 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-data\") pod \"7cba5790-7577-462a-aec0-759ff1d6bb40\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " Oct 30 19:46:22.854525 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.854503 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-certs\") pod \"7cba5790-7577-462a-aec0-759ff1d6bb40\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " Oct 30 19:46:22.854679 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.854611 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-envoy\") pod \"7cba5790-7577-462a-aec0-759ff1d6bb40\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " Oct 30 19:46:22.854679 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.854642 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-socket\") pod \"7cba5790-7577-462a-aec0-759ff1d6bb40\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " Oct 30 19:46:22.854789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.854707 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-podinfo\") pod \"7cba5790-7577-462a-aec0-759ff1d6bb40\" (UID: \"7cba5790-7577-462a-aec0-759ff1d6bb40\") " Oct 30 19:46:22.854789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.854757 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-credential-socket" pod "7cba5790-7577-462a-aec0-759ff1d6bb40" (UID: "7cba5790-7577-462a-aec0-759ff1d6bb40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:22.855040 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.855011 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cba5790-7577-462a-aec0-759ff1d6bb40-istiod-ca-cert" pod "7cba5790-7577-462a-aec0-759ff1d6bb40" (UID: "7cba5790-7577-462a-aec0-759ff1d6bb40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:22.855302 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.855272 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-data" pod "7cba5790-7577-462a-aec0-759ff1d6bb40" (UID: "7cba5790-7577-462a-aec0-759ff1d6bb40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:22.855302 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.855296 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cba5790-7577-462a-aec0-759ff1d6bb40-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:22.855441 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.855317 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:22.855441 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.855291 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-socket" pod "7cba5790-7577-462a-aec0-759ff1d6bb40" (UID: "7cba5790-7577-462a-aec0-759ff1d6bb40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:22.855602 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.855582 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-certs" pod "7cba5790-7577-462a-aec0-759ff1d6bb40" (UID: "7cba5790-7577-462a-aec0-759ff1d6bb40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:22.857146 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.857116 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-token" pod "7cba5790-7577-462a-aec0-759ff1d6bb40" (UID: "7cba5790-7577-462a-aec0-759ff1d6bb40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:22.857399 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.857252 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-kube-api-access-f9rzc" pod "7cba5790-7577-462a-aec0-759ff1d6bb40" (UID: "7cba5790-7577-462a-aec0-759ff1d6bb40"). InnerVolumeSpecName "kube-api-access-f9rzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:22.857566 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.857517 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-podinfo" pod "7cba5790-7577-462a-aec0-759ff1d6bb40" (UID: "7cba5790-7577-462a-aec0-759ff1d6bb40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:22.857907 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.857885 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-envoy" pod "7cba5790-7577-462a-aec0-759ff1d6bb40" (UID: "7cba5790-7577-462a-aec0-759ff1d6bb40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:22.957096 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.957063 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:22.957096 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.957089 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9rzc\" (UniqueName: \"kubernetes.io/projected/7cba5790-7577-462a-aec0-759ff1d6bb40-kube-api-access-f9rzc\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:22.957366 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.957127 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:22.957366 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.957138 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:22.957366 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.957145 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:22.957366 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.957152 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cba5790-7577-462a-aec0-759ff1d6bb40-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:22.957366 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:22.957160 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cba5790-7577-462a-aec0-759ff1d6bb40-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:23.467981 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:23.467937 4904 generic.go:356] "Generic (PLEG): container finished" podID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerID="405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a" exitCode=0 Oct 30 19:46:23.468217 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:23.468194 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" event={"ID":"7cba5790-7577-462a-aec0-759ff1d6bb40","Type":"ContainerDied","Data":"405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a"} Oct 30 19:46:23.468307 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:23.468291 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" event={"ID":"7cba5790-7577-462a-aec0-759ff1d6bb40","Type":"ContainerDied","Data":"01571ce9ec9350b244dfe2ee5601a2988b9a133ef9f4d1081c4d0159911970bb"} Oct 30 19:46:23.468371 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:23.468339 4904 scope.go:122] "RemoveContainer" containerID="405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a" Oct 30 19:46:23.468580 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:23.468562 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m" Oct 30 19:46:23.482232 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:23.482208 4904 scope.go:122] "RemoveContainer" containerID="405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a" Oct 30 19:46:23.482514 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:46:23.482494 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a\": container with ID starting with 405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a not found: ID does not exist" containerID="405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a" Oct 30 19:46:23.482580 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:23.482523 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a"} err="failed to get container status \"405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a\": rpc error: code = NotFound desc = could not find container \"405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a\": container with ID starting with 405209765db63e3c83cd36ad5753bab97615fb503bdd14c0fca088a4d376322a not found: ID does not exist" Oct 30 19:46:23.540305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:23.540246 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m"] Oct 30 19:46:23.552846 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:23.552820 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-hf79m"] Oct 30 19:46:24.509904 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:46:24.509870 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:46:25.224862 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:25.224832 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" path="/var/lib/kubelet/pods/7cba5790-7577-462a-aec0-759ff1d6bb40/volumes" Oct 30 19:46:26.754984 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.754924 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:46:26.803034 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803006 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-podinfo\") pod \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " Oct 30 19:46:26.803202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803052 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-data\") pod \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " Oct 30 19:46:26.803202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803086 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-socket\") pod \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " Oct 30 19:46:26.803202 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803169 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-token\" (UniqueName: \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-token\") pod \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " Oct 30 19:46:26.803383 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803209 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istiod-ca-cert\") pod \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " Oct 30 19:46:26.803383 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803243 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-credential-socket\") pod \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " Oct 30 19:46:26.803485 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803444 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-data" pod "6e21f26d-3a3b-4913-8a9b-0f6801f53456" (UID: "6e21f26d-3a3b-4913-8a9b-0f6801f53456"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.803538 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803455 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-socket" pod "6e21f26d-3a3b-4913-8a9b-0f6801f53456" (UID: "6e21f26d-3a3b-4913-8a9b-0f6801f53456"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.803538 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803449 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-kube-api-access-6rsq4\" (UniqueName: \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-kube-api-access-6rsq4\") pod \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " Oct 30 19:46:26.803640 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803554 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-credential-socket" pod "6e21f26d-3a3b-4913-8a9b-0f6801f53456" (UID: "6e21f26d-3a3b-4913-8a9b-0f6801f53456"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.803695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803644 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istiod-ca-cert" pod "6e21f26d-3a3b-4913-8a9b-0f6801f53456" (UID: "6e21f26d-3a3b-4913-8a9b-0f6801f53456"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:26.803695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803660 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-envoy\") pod \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " Oct 30 19:46:26.803806 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.803736 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-certs\") pod \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\" (UID: \"6e21f26d-3a3b-4913-8a9b-0f6801f53456\") " Oct 30 19:46:26.804246 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.804222 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-certs" pod "6e21f26d-3a3b-4913-8a9b-0f6801f53456" (UID: "6e21f26d-3a3b-4913-8a9b-0f6801f53456"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.804332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.804230 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:26.804332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.804288 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:26.804332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.804303 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:26.804332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.804319 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:26.805785 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.805765 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-envoy" pod "6e21f26d-3a3b-4913-8a9b-0f6801f53456" (UID: "6e21f26d-3a3b-4913-8a9b-0f6801f53456"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.806077 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.806038 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-kube-api-access-6rsq4" pod "6e21f26d-3a3b-4913-8a9b-0f6801f53456" (UID: "6e21f26d-3a3b-4913-8a9b-0f6801f53456"). InnerVolumeSpecName "kube-api-access-6rsq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:26.806224 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.806196 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-token" pod "6e21f26d-3a3b-4913-8a9b-0f6801f53456" (UID: "6e21f26d-3a3b-4913-8a9b-0f6801f53456"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:26.806294 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.806251 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-podinfo" pod "6e21f26d-3a3b-4913-8a9b-0f6801f53456" (UID: "6e21f26d-3a3b-4913-8a9b-0f6801f53456"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:26.906015 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.905968 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:26.906015 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.906009 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e21f26d-3a3b-4913-8a9b-0f6801f53456-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:26.906296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.906026 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:26.906296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.906039 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:26.906296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:26.906052 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rsq4\" (UniqueName: \"kubernetes.io/projected/6e21f26d-3a3b-4913-8a9b-0f6801f53456-kube-api-access-6rsq4\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:27.504361 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:27.504274 4904 generic.go:356] "Generic (PLEG): container finished" podID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerID="f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7" exitCode=0 Oct 30 19:46:27.504361 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:27.504321 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" Oct 30 19:46:27.504592 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:27.504353 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" event={"ID":"6e21f26d-3a3b-4913-8a9b-0f6801f53456","Type":"ContainerDied","Data":"f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7"} Oct 30 19:46:27.504592 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:27.504422 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg" event={"ID":"6e21f26d-3a3b-4913-8a9b-0f6801f53456","Type":"ContainerDied","Data":"38ac11e72eb214d6b0b59b2b4c6b32c3334506515d4ae9668e30d8010985daca"} Oct 30 19:46:27.504592 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:27.504453 4904 scope.go:122] "RemoveContainer" containerID="f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7" Oct 30 19:46:27.515296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:27.515279 4904 scope.go:122] "RemoveContainer" containerID="f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7" Oct 30 19:46:27.515589 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:46:27.515562 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7\": container with ID starting with f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7 not found: ID does not exist" containerID="f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7" Oct 30 19:46:27.515664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:27.515593 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7"} err="failed to get container status \"f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7\": rpc error: code = NotFound desc = could not find container \"f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7\": container with ID starting with f8eb723d9fa773ba408a6c60595ccfb88d3cc846c4b511c506dd9b68239223a7 not found: ID does not exist" Oct 30 19:46:27.583561 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:27.583535 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg"] Oct 30 19:46:27.598148 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:27.598127 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-jrmdg"] Oct 30 19:46:29.239840 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:29.239804 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" path="/var/lib/kubelet/pods/6e21f26d-3a3b-4913-8a9b-0f6801f53456/volumes" Oct 30 19:46:32.509850 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:32.509817 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh"] Oct 30 19:46:32.510284 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:32.510008 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" containerID="cri-o://6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b" gracePeriod=30 Oct 30 19:46:37.822367 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.822334 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:46:37.883874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.883839 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-istio-token\" (UniqueName: \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-istio-token\") pod \"059991be-f076-488f-9aa1-70521ef6460a\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " Oct 30 19:46:37.884678 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.884639 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-kube-api-access-t5nbc\" (UniqueName: \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-kube-api-access-t5nbc\") pod \"059991be-f076-488f-9aa1-70521ef6460a\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " Oct 30 19:46:37.928132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.927745 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-socket\") pod \"059991be-f076-488f-9aa1-70521ef6460a\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " Oct 30 19:46:37.928132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.927879 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-envoy\") pod \"059991be-f076-488f-9aa1-70521ef6460a\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " Oct 30 19:46:37.928132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.927942 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-certs\") pod \"059991be-f076-488f-9aa1-70521ef6460a\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " Oct 30 19:46:37.928132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.927991 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-credential-socket\") pod \"059991be-f076-488f-9aa1-70521ef6460a\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " Oct 30 19:46:37.928132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.928018 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059991be-f076-488f-9aa1-70521ef6460a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059991be-f076-488f-9aa1-70521ef6460a-istiod-ca-cert\") pod \"059991be-f076-488f-9aa1-70521ef6460a\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " Oct 30 19:46:37.928132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.928045 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-data\") pod \"059991be-f076-488f-9aa1-70521ef6460a\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " Oct 30 19:46:37.928132 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.928134 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/059991be-f076-488f-9aa1-70521ef6460a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059991be-f076-488f-9aa1-70521ef6460a-istio-podinfo\") pod \"059991be-f076-488f-9aa1-70521ef6460a\" (UID: \"059991be-f076-488f-9aa1-70521ef6460a\") " Oct 30 19:46:37.931661 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.929787 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-istio-token" pod "059991be-f076-488f-9aa1-70521ef6460a" (UID: "059991be-f076-488f-9aa1-70521ef6460a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:37.931661 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.930957 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-credential-socket" pod "059991be-f076-488f-9aa1-70521ef6460a" (UID: "059991be-f076-488f-9aa1-70521ef6460a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:37.931661 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.931194 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-certs" pod "059991be-f076-488f-9aa1-70521ef6460a" (UID: "059991be-f076-488f-9aa1-70521ef6460a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:37.931661 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.931589 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059991be-f076-488f-9aa1-70521ef6460a-istiod-ca-cert" pod "059991be-f076-488f-9aa1-70521ef6460a" (UID: "059991be-f076-488f-9aa1-70521ef6460a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:37.932076 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.932020 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-data" pod "059991be-f076-488f-9aa1-70521ef6460a" (UID: "059991be-f076-488f-9aa1-70521ef6460a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:37.935332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.932548 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-socket" pod "059991be-f076-488f-9aa1-70521ef6460a" (UID: "059991be-f076-488f-9aa1-70521ef6460a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:37.935332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.934542 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/059991be-f076-488f-9aa1-70521ef6460a-istio-podinfo" pod "059991be-f076-488f-9aa1-70521ef6460a" (UID: "059991be-f076-488f-9aa1-70521ef6460a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:37.935332 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.934741 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-kube-api-access-t5nbc" pod "059991be-f076-488f-9aa1-70521ef6460a" (UID: "059991be-f076-488f-9aa1-70521ef6460a"). InnerVolumeSpecName "kube-api-access-t5nbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:37.952325 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:37.952012 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-envoy" pod "059991be-f076-488f-9aa1-70521ef6460a" (UID: "059991be-f076-488f-9aa1-70521ef6460a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:38.030607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.029883 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:38.030607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.029921 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:38.030607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.029935 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:38.030607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.029949 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059991be-f076-488f-9aa1-70521ef6460a-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:38.030607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.029961 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:38.030607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.029974 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059991be-f076-488f-9aa1-70521ef6460a-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:38.030607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.029987 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:38.030607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.030000 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5nbc\" (UniqueName: \"kubernetes.io/projected/059991be-f076-488f-9aa1-70521ef6460a-kube-api-access-t5nbc\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:38.030607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.030015 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059991be-f076-488f-9aa1-70521ef6460a-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:38.624153 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.624120 4904 generic.go:356] "Generic (PLEG): container finished" podID="059991be-f076-488f-9aa1-70521ef6460a" containerID="6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b" exitCode=0 Oct 30 19:46:38.624306 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.624181 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" Oct 30 19:46:38.624306 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.624208 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" event={"ID":"059991be-f076-488f-9aa1-70521ef6460a","Type":"ContainerDied","Data":"6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b"} Oct 30 19:46:38.624306 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.624266 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh" event={"ID":"059991be-f076-488f-9aa1-70521ef6460a","Type":"ContainerDied","Data":"5f8aa4805edd7f76a7ffbd9895fae48fe9eb7091197e28bc2882c6323a3c45b4"} Oct 30 19:46:38.624306 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.624288 4904 scope.go:122] "RemoveContainer" containerID="6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b" Oct 30 19:46:38.638281 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.638263 4904 scope.go:122] "RemoveContainer" containerID="6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b" Oct 30 19:46:38.638672 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:46:38.638653 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b\": container with ID starting with 6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b not found: ID does not exist" containerID="6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b" Oct 30 19:46:38.638771 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.638684 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b"} err="failed to get container status \"6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b\": rpc error: code = NotFound desc = could not find container \"6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b\": container with ID starting with 6cb70ecf582ee9c2afb325c2aade4317647732800a5359f3f39ebf0af61c951b not found: ID does not exist" Oct 30 19:46:38.698112 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.698072 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh"] Oct 30 19:46:38.701271 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:38.701251 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-wjtqh"] Oct 30 19:46:39.224308 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:39.224280 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059991be-f076-488f-9aa1-70521ef6460a" path="/var/lib/kubelet/pods/059991be-f076-488f-9aa1-70521ef6460a/volumes" Oct 30 19:46:49.256656 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:49.256226 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx"] Oct 30 19:46:49.256656 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:49.256473 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerName="istio-proxy" containerID="cri-o://a9b4a0c414648ee590b2ca613e85cd137ab1bda437bcd26c04eb0c1ef4b9fba1" gracePeriod=30 Oct 30 19:46:49.283135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:49.280589 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g"] Oct 30 19:46:49.283135 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:49.280815 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" containerID="cri-o://3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25" gracePeriod=30 Oct 30 19:46:52.423345 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.423304 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz"] Oct 30 19:46:52.424043 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.423598 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" podUID="8d727b1d-8115-4618-9d22-2c320c1c9800" containerName="maas-api" containerID="cri-o://9b9bb7ff7d8a21b47f697988a668e3c12895623380509226cff2f1d521a09ee0" gracePeriod=30 Oct 30 19:46:52.775397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.774844 4904 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:46:52.775397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.774921 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:46:52.780528 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.780208 4904 generic.go:356] "Generic (PLEG): container finished" podID="8d727b1d-8115-4618-9d22-2c320c1c9800" containerID="9b9bb7ff7d8a21b47f697988a668e3c12895623380509226cff2f1d521a09ee0" exitCode=0 Oct 30 19:46:52.780528 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.780460 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" event={"ID":"8d727b1d-8115-4618-9d22-2c320c1c9800","Type":"ContainerDied","Data":"9b9bb7ff7d8a21b47f697988a668e3c12895623380509226cff2f1d521a09ee0"} Oct 30 19:46:52.848372 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.848338 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:46:52.974326 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.974297 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d727b1d-8115-4618-9d22-2c320c1c9800-kube-api-access-rwpjn\" (UniqueName: \"kubernetes.io/projected/8d727b1d-8115-4618-9d22-2c320c1c9800-kube-api-access-rwpjn\") pod \"8d727b1d-8115-4618-9d22-2c320c1c9800\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " Oct 30 19:46:52.974511 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.974343 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d727b1d-8115-4618-9d22-2c320c1c9800-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d727b1d-8115-4618-9d22-2c320c1c9800-openshift-service-ca\") pod \"8d727b1d-8115-4618-9d22-2c320c1c9800\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " Oct 30 19:46:52.974511 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.974382 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-metrics-certs\") pod \"8d727b1d-8115-4618-9d22-2c320c1c9800\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " Oct 30 19:46:52.974644 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.974553 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-maas-api-tls\") pod \"8d727b1d-8115-4618-9d22-2c320c1c9800\" (UID: \"8d727b1d-8115-4618-9d22-2c320c1c9800\") " Oct 30 19:46:52.974728 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.974691 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d727b1d-8115-4618-9d22-2c320c1c9800-openshift-service-ca" pod "8d727b1d-8115-4618-9d22-2c320c1c9800" (UID: "8d727b1d-8115-4618-9d22-2c320c1c9800"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:52.975472 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.975444 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d727b1d-8115-4618-9d22-2c320c1c9800-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:52.976743 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.976717 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-metrics-certs" pod "8d727b1d-8115-4618-9d22-2c320c1c9800" (UID: "8d727b1d-8115-4618-9d22-2c320c1c9800"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:52.977255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.976797 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d727b1d-8115-4618-9d22-2c320c1c9800-kube-api-access-rwpjn" pod "8d727b1d-8115-4618-9d22-2c320c1c9800" (UID: "8d727b1d-8115-4618-9d22-2c320c1c9800"). InnerVolumeSpecName "kube-api-access-rwpjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:52.977255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:52.976810 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-maas-api-tls" pod "8d727b1d-8115-4618-9d22-2c320c1c9800" (UID: "8d727b1d-8115-4618-9d22-2c320c1c9800"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:53.076433 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:53.076320 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwpjn\" (UniqueName: \"kubernetes.io/projected/8d727b1d-8115-4618-9d22-2c320c1c9800-kube-api-access-rwpjn\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:53.076433 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:53.076351 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:53.076433 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:53.076363 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d727b1d-8115-4618-9d22-2c320c1c9800-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:53.788910 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:53.788875 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" event={"ID":"8d727b1d-8115-4618-9d22-2c320c1c9800","Type":"ContainerDied","Data":"841d60993a1a6b499433a5c38dc19be35dab50110c5c1380688b0b31cc3baea5"} Oct 30 19:46:53.788910 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:53.788896 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz" Oct 30 19:46:53.789417 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:53.788946 4904 scope.go:122] "RemoveContainer" containerID="9b9bb7ff7d8a21b47f697988a668e3c12895623380509226cff2f1d521a09ee0" Oct 30 19:46:53.847270 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:53.847243 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz"] Oct 30 19:46:53.860495 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:53.860474 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe4ec3-b55c8f957-gncdz"] Oct 30 19:46:54.519947 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:46:54.519909 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:46:54.577722 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.577698 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:46:54.696153 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696086 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-kube-api-access-jnwwd\" (UniqueName: \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-kube-api-access-jnwwd\") pod \"3793771e-d940-4326-b09d-288113928d46\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " Oct 30 19:46:54.696329 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696217 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-socket\") pod \"3793771e-d940-4326-b09d-288113928d46\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " Oct 30 19:46:54.696329 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696271 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-data\") pod \"3793771e-d940-4326-b09d-288113928d46\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " Oct 30 19:46:54.696465 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696369 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3793771e-d940-4326-b09d-288113928d46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3793771e-d940-4326-b09d-288113928d46-istio-podinfo\") pod \"3793771e-d940-4326-b09d-288113928d46\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " Oct 30 19:46:54.696465 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696417 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-credential-socket\") pod \"3793771e-d940-4326-b09d-288113928d46\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " Oct 30 19:46:54.696573 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696510 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3793771e-d940-4326-b09d-288113928d46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3793771e-d940-4326-b09d-288113928d46-istiod-ca-cert\") pod \"3793771e-d940-4326-b09d-288113928d46\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " Oct 30 19:46:54.696573 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696538 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-data" pod "3793771e-d940-4326-b09d-288113928d46" (UID: "3793771e-d940-4326-b09d-288113928d46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:54.696573 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696548 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-socket" pod "3793771e-d940-4326-b09d-288113928d46" (UID: "3793771e-d940-4326-b09d-288113928d46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:54.696573 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696564 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-envoy\") pod \"3793771e-d940-4326-b09d-288113928d46\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " Oct 30 19:46:54.696785 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696605 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-certs\") pod \"3793771e-d940-4326-b09d-288113928d46\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " Oct 30 19:46:54.696785 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696655 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-istio-token\" (UniqueName: \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-istio-token\") pod \"3793771e-d940-4326-b09d-288113928d46\" (UID: \"3793771e-d940-4326-b09d-288113928d46\") " Oct 30 19:46:54.696785 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696745 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-credential-socket" pod "3793771e-d940-4326-b09d-288113928d46" (UID: "3793771e-d940-4326-b09d-288113928d46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:54.696931 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.696851 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3793771e-d940-4326-b09d-288113928d46-istiod-ca-cert" pod "3793771e-d940-4326-b09d-288113928d46" (UID: "3793771e-d940-4326-b09d-288113928d46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:54.697191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.697165 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-certs" pod "3793771e-d940-4326-b09d-288113928d46" (UID: "3793771e-d940-4326-b09d-288113928d46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:54.697416 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.697394 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3793771e-d940-4326-b09d-288113928d46-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:54.697683 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.697421 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:54.697683 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.697518 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:54.697683 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.697608 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:54.697683 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.697622 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:54.698972 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.698950 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3793771e-d940-4326-b09d-288113928d46-istio-podinfo" pod "3793771e-d940-4326-b09d-288113928d46" (UID: "3793771e-d940-4326-b09d-288113928d46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:54.699185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.699160 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-envoy" pod "3793771e-d940-4326-b09d-288113928d46" (UID: "3793771e-d940-4326-b09d-288113928d46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:54.699185 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.699169 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-kube-api-access-jnwwd" pod "3793771e-d940-4326-b09d-288113928d46" (UID: "3793771e-d940-4326-b09d-288113928d46"). InnerVolumeSpecName "kube-api-access-jnwwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:54.699450 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.699434 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-istio-token" pod "3793771e-d940-4326-b09d-288113928d46" (UID: "3793771e-d940-4326-b09d-288113928d46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:54.798560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.798530 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3793771e-d940-4326-b09d-288113928d46-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:54.798560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.798552 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3793771e-d940-4326-b09d-288113928d46-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:54.798560 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.798561 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:54.799085 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.798569 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnwwd\" (UniqueName: \"kubernetes.io/projected/3793771e-d940-4326-b09d-288113928d46-kube-api-access-jnwwd\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:54.799085 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.798718 4904 generic.go:356] "Generic (PLEG): container finished" podID="3793771e-d940-4326-b09d-288113928d46" containerID="3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25" exitCode=0 Oct 30 19:46:54.799085 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.798790 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" Oct 30 19:46:54.799085 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.798800 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" event={"ID":"3793771e-d940-4326-b09d-288113928d46","Type":"ContainerDied","Data":"3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25"} Oct 30 19:46:54.799085 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.798850 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g" event={"ID":"3793771e-d940-4326-b09d-288113928d46","Type":"ContainerDied","Data":"17af1a86d380b38fa82950cb9be4a799bd35aa3e0f21f0fead1ff52ffdee72d2"} Oct 30 19:46:54.799085 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.798872 4904 scope.go:122] "RemoveContainer" containerID="3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25" Oct 30 19:46:54.801203 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.801175 4904 generic.go:356] "Generic (PLEG): container finished" podID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerID="a9b4a0c414648ee590b2ca613e85cd137ab1bda437bcd26c04eb0c1ef4b9fba1" exitCode=0 Oct 30 19:46:54.801319 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.801247 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" event={"ID":"46da1cc6-92fa-47a0-91a8-eaf26cd86427","Type":"ContainerDied","Data":"a9b4a0c414648ee590b2ca613e85cd137ab1bda437bcd26c04eb0c1ef4b9fba1"} Oct 30 19:46:54.810580 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.810542 4904 scope.go:122] "RemoveContainer" containerID="3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25" Oct 30 19:46:54.810873 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:46:54.810853 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25\": container with ID starting with 3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25 not found: ID does not exist" containerID="3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25" Oct 30 19:46:54.810944 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.810881 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25"} err="failed to get container status \"3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25\": rpc error: code = NotFound desc = could not find container \"3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25\": container with ID starting with 3ad43301df8cb45f004de2bed34aeee2034ad9eb07e4b2862e969fe679bebe25 not found: ID does not exist" Oct 30 19:46:54.858516 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.858489 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g"] Oct 30 19:46:54.874336 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.874300 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-8xn7g"] Oct 30 19:46:54.979379 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:54.979357 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:46:55.104013 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.103933 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-envoy\") pod \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " Oct 30 19:46:55.104315 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.104285 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-kube-api-access-9cf6p\" (UniqueName: \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-kube-api-access-9cf6p\") pod \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " Oct 30 19:46:55.104712 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.104692 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-token\" (UniqueName: \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-token\") pod \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " Oct 30 19:46:55.105027 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.105013 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istiod-ca-cert\") pod \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " Oct 30 19:46:55.105255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.105239 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-credential-socket\") pod \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " Oct 30 19:46:55.105400 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.105387 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-podinfo\") pod \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " Oct 30 19:46:55.105881 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.105521 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-credential-socket" pod "46da1cc6-92fa-47a0-91a8-eaf26cd86427" (UID: "46da1cc6-92fa-47a0-91a8-eaf26cd86427"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:55.105961 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.105699 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istiod-ca-cert" pod "46da1cc6-92fa-47a0-91a8-eaf26cd86427" (UID: "46da1cc6-92fa-47a0-91a8-eaf26cd86427"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:55.106076 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.106059 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-data\") pod \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " Oct 30 19:46:55.106533 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.106513 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-socket\") pod \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " Oct 30 19:46:55.106729 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.106715 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-certs\") pod \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\" (UID: \"46da1cc6-92fa-47a0-91a8-eaf26cd86427\") " Oct 30 19:46:55.107265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.106769 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-data" pod "46da1cc6-92fa-47a0-91a8-eaf26cd86427" (UID: "46da1cc6-92fa-47a0-91a8-eaf26cd86427"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:55.107265 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.107207 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-socket" pod "46da1cc6-92fa-47a0-91a8-eaf26cd86427" (UID: "46da1cc6-92fa-47a0-91a8-eaf26cd86427"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:55.107427 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.107383 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-certs" pod "46da1cc6-92fa-47a0-91a8-eaf26cd86427" (UID: "46da1cc6-92fa-47a0-91a8-eaf26cd86427"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:55.107487 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.107438 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-kube-api-access-9cf6p" pod "46da1cc6-92fa-47a0-91a8-eaf26cd86427" (UID: "46da1cc6-92fa-47a0-91a8-eaf26cd86427"). InnerVolumeSpecName "kube-api-access-9cf6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:55.107487 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.107469 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-token" pod "46da1cc6-92fa-47a0-91a8-eaf26cd86427" (UID: "46da1cc6-92fa-47a0-91a8-eaf26cd86427"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:55.107647 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.107481 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-envoy" pod "46da1cc6-92fa-47a0-91a8-eaf26cd86427" (UID: "46da1cc6-92fa-47a0-91a8-eaf26cd86427"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:55.107832 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.107809 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-podinfo" pod "46da1cc6-92fa-47a0-91a8-eaf26cd86427" (UID: "46da1cc6-92fa-47a0-91a8-eaf26cd86427"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:55.108019 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.107997 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:55.108084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.108023 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:55.108084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.108039 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:55.108084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.108050 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:55.108084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.108063 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:55.108084 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.108075 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:55.108288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.108087 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cf6p\" (UniqueName: \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-kube-api-access-9cf6p\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:55.108288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.108118 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:55.108288 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.108132 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46da1cc6-92fa-47a0-91a8-eaf26cd86427-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:46:55.222369 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.222341 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3793771e-d940-4326-b09d-288113928d46" path="/var/lib/kubelet/pods/3793771e-d940-4326-b09d-288113928d46/volumes" Oct 30 19:46:55.222838 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.222817 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d727b1d-8115-4618-9d22-2c320c1c9800" path="/var/lib/kubelet/pods/8d727b1d-8115-4618-9d22-2c320c1c9800/volumes" Oct 30 19:46:55.820902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.820871 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" Oct 30 19:46:55.820902 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.820891 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx" event={"ID":"46da1cc6-92fa-47a0-91a8-eaf26cd86427","Type":"ContainerDied","Data":"a86c13de3021f60797b039fe720f3e2201d4af948d5f77d1d0bccf2f5c010bf9"} Oct 30 19:46:55.821381 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.820980 4904 scope.go:122] "RemoveContainer" containerID="a9b4a0c414648ee590b2ca613e85cd137ab1bda437bcd26c04eb0c1ef4b9fba1" Oct 30 19:46:55.892432 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.892406 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx"] Oct 30 19:46:55.905944 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:55.905916 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-m96gx"] Oct 30 19:46:57.223086 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:46:57.223045 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" path="/var/lib/kubelet/pods/46da1cc6-92fa-47a0-91a8-eaf26cd86427/volumes" Oct 30 19:47:05.530434 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:05.530403 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq"] Oct 30 19:47:05.531035 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:05.530596 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" podUID="61a7275c-1864-4f89-98c7-cef6efebe04d" containerName="maas-api" containerID="cri-o://b77a67ca9c9e9447ca725ef081cf3f76cdf4864796861b64060c9b3959c1ef13" gracePeriod=30 Oct 30 19:47:05.915890 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:05.915640 4904 generic.go:356] "Generic (PLEG): container finished" podID="61a7275c-1864-4f89-98c7-cef6efebe04d" containerID="b77a67ca9c9e9447ca725ef081cf3f76cdf4864796861b64060c9b3959c1ef13" exitCode=0 Oct 30 19:47:05.915890 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:05.915811 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" event={"ID":"61a7275c-1864-4f89-98c7-cef6efebe04d","Type":"ContainerDied","Data":"b77a67ca9c9e9447ca725ef081cf3f76cdf4864796861b64060c9b3959c1ef13"} Oct 30 19:47:05.927208 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:05.925675 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:47:05.987126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:05.983652 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-metrics-certs\") pod \"61a7275c-1864-4f89-98c7-cef6efebe04d\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " Oct 30 19:47:05.987126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:05.983717 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-maas-api-tls\") pod \"61a7275c-1864-4f89-98c7-cef6efebe04d\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " Oct 30 19:47:05.987126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:05.983817 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a7275c-1864-4f89-98c7-cef6efebe04d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a7275c-1864-4f89-98c7-cef6efebe04d-openshift-service-ca\") pod \"61a7275c-1864-4f89-98c7-cef6efebe04d\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " Oct 30 19:47:05.987126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:05.983932 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a7275c-1864-4f89-98c7-cef6efebe04d-kube-api-access-kd9hm\" (UniqueName: \"kubernetes.io/projected/61a7275c-1864-4f89-98c7-cef6efebe04d-kube-api-access-kd9hm\") pod \"61a7275c-1864-4f89-98c7-cef6efebe04d\" (UID: \"61a7275c-1864-4f89-98c7-cef6efebe04d\") " Oct 30 19:47:06.011126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.000684 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-metrics-certs" pod "61a7275c-1864-4f89-98c7-cef6efebe04d" (UID: "61a7275c-1864-4f89-98c7-cef6efebe04d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:06.011126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.000820 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a7275c-1864-4f89-98c7-cef6efebe04d-kube-api-access-kd9hm" pod "61a7275c-1864-4f89-98c7-cef6efebe04d" (UID: "61a7275c-1864-4f89-98c7-cef6efebe04d"). InnerVolumeSpecName "kube-api-access-kd9hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:06.011126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.001525 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-maas-api-tls" pod "61a7275c-1864-4f89-98c7-cef6efebe04d" (UID: "61a7275c-1864-4f89-98c7-cef6efebe04d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:06.011126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.010315 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a7275c-1864-4f89-98c7-cef6efebe04d-openshift-service-ca" pod "61a7275c-1864-4f89-98c7-cef6efebe04d" (UID: "61a7275c-1864-4f89-98c7-cef6efebe04d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:06.085906 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.085873 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a7275c-1864-4f89-98c7-cef6efebe04d-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:06.085906 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.085905 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd9hm\" (UniqueName: \"kubernetes.io/projected/61a7275c-1864-4f89-98c7-cef6efebe04d-kube-api-access-kd9hm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:06.086128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.085916 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:06.086128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.085925 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a7275c-1864-4f89-98c7-cef6efebe04d-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:06.923958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.923917 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" event={"ID":"61a7275c-1864-4f89-98c7-cef6efebe04d","Type":"ContainerDied","Data":"9944656729f262c76be8c6ecfb3f095a94edc36574bca7900cb0dddb51c488cc"} Oct 30 19:47:06.923958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.923957 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq" Oct 30 19:47:06.924501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:06.924021 4904 scope.go:122] "RemoveContainer" containerID="b77a67ca9c9e9447ca725ef081cf3f76cdf4864796861b64060c9b3959c1ef13" Oct 30 19:47:07.012705 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:07.012680 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq"] Oct 30 19:47:07.025766 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:07.025738 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b4c34-5b549d9677-cs6gq"] Oct 30 19:47:07.223883 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:07.223810 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a7275c-1864-4f89-98c7-cef6efebe04d" path="/var/lib/kubelet/pods/61a7275c-1864-4f89-98c7-cef6efebe04d/volumes" Oct 30 19:47:14.937651 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:14.937605 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq"] Oct 30 19:47:14.938553 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:14.938489 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerName="istio-proxy" containerID="cri-o://e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c" gracePeriod=30 Oct 30 19:47:14.956212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:14.946143 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x"] Oct 30 19:47:14.956212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:14.946355 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" containerID="cri-o://28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da" gracePeriod=30 Oct 30 19:47:14.956212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:14.950428 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49"] Oct 30 19:47:14.956212 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:14.950600 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" containerID="cri-o://0e181c0b871d188c29da9e93f01d0543d5aa0bf9869dbde5dd4413b69ef30ee3" gracePeriod=30 Oct 30 19:47:17.750360 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:17.750324 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z"] Oct 30 19:47:17.750755 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:17.750592 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" containerID="cri-o://ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651" gracePeriod=30 Oct 30 19:47:19.264051 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:19.264009 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2"] Oct 30 19:47:19.278279 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:19.278253 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5"] Oct 30 19:47:20.147013 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.146981 4904 generic.go:356] "Generic (PLEG): container finished" podID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerID="0e181c0b871d188c29da9e93f01d0543d5aa0bf9869dbde5dd4413b69ef30ee3" exitCode=0 Oct 30 19:47:20.147182 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.147053 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" event={"ID":"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2","Type":"ContainerDied","Data":"0e181c0b871d188c29da9e93f01d0543d5aa0bf9869dbde5dd4413b69ef30ee3"} Oct 30 19:47:20.275076 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.273794 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:47:20.308133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.307422 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-certs\") pod \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " Oct 30 19:47:20.308133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.307492 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-credential-socket\") pod \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " Oct 30 19:47:20.308133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.307540 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istiod-ca-cert\") pod \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " Oct 30 19:47:20.308133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.307640 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-token\" (UniqueName: \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-token\") pod \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " Oct 30 19:47:20.308133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.307699 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-data\") pod \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " Oct 30 19:47:20.308133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.307754 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-certs" pod "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" (UID: "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.308133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.307761 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-podinfo\") pod \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " Oct 30 19:47:20.308133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.307858 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-kube-api-access-9wcjw\" (UniqueName: \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-kube-api-access-9wcjw\") pod \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " Oct 30 19:47:20.308133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.307898 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-envoy\") pod \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " Oct 30 19:47:20.308133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.307964 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-socket\") pod \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\" (UID: \"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2\") " Oct 30 19:47:20.308779 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.308665 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.309764 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.309513 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-socket" pod "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" (UID: "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.311305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.310923 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-data" pod "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" (UID: "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.311305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.311165 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-credential-socket" pod "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" (UID: "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.311305 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.311277 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-podinfo" pod "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" (UID: "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:47:20.315083 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.315047 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-kube-api-access-9wcjw" pod "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" (UID: "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2"). InnerVolumeSpecName "kube-api-access-9wcjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:20.315198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.315146 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istiod-ca-cert" pod "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" (UID: "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:20.315281 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.315263 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-envoy" pod "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" (UID: "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.315472 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.315450 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-token" pod "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" (UID: "08c3a4f0-8862-43a6-89ac-3e6c847a6ab2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:20.409884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.409854 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.409884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.409882 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.409884 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.409892 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.410198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.409901 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wcjw\" (UniqueName: \"kubernetes.io/projected/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-kube-api-access-9wcjw\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.410198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.409914 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.410198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.409923 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.410198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.409930 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.410198 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.409938 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.415789 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.415769 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:47:20.419742 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.419723 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:47:20.510858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.510826 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-envoy\") pod \"c103d769-22e3-4f04-92f6-fc9677f3098b\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " Oct 30 19:47:20.511036 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.510873 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-kube-api-access-dk4x6\" (UniqueName: \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-kube-api-access-dk4x6\") pod \"c103d769-22e3-4f04-92f6-fc9677f3098b\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " Oct 30 19:47:20.511036 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.510920 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-data\") pod \"a5e31c74-47bc-453f-81ef-795c60afeb3c\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " Oct 30 19:47:20.511036 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.510960 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-kube-api-access-c8m2r\" (UniqueName: \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-kube-api-access-c8m2r\") pod \"a5e31c74-47bc-453f-81ef-795c60afeb3c\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " Oct 30 19:47:20.511036 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511010 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-certs\") pod \"c103d769-22e3-4f04-92f6-fc9677f3098b\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " Oct 30 19:47:20.511290 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511058 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-podinfo\") pod \"a5e31c74-47bc-453f-81ef-795c60afeb3c\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " Oct 30 19:47:20.511290 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511191 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-token\" (UniqueName: \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-token\") pod \"a5e31c74-47bc-453f-81ef-795c60afeb3c\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " Oct 30 19:47:20.511290 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511222 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-data\") pod \"c103d769-22e3-4f04-92f6-fc9677f3098b\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " Oct 30 19:47:20.511290 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511251 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-envoy\") pod \"a5e31c74-47bc-453f-81ef-795c60afeb3c\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " Oct 30 19:47:20.511487 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511293 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-credential-socket\") pod \"a5e31c74-47bc-453f-81ef-795c60afeb3c\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " Oct 30 19:47:20.511487 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511339 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c103d769-22e3-4f04-92f6-fc9677f3098b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c103d769-22e3-4f04-92f6-fc9677f3098b-istiod-ca-cert\") pod \"c103d769-22e3-4f04-92f6-fc9677f3098b\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " Oct 30 19:47:20.511487 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511372 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e31c74-47bc-453f-81ef-795c60afeb3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e31c74-47bc-453f-81ef-795c60afeb3c-istiod-ca-cert\") pod \"a5e31c74-47bc-453f-81ef-795c60afeb3c\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " Oct 30 19:47:20.511638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511491 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-socket\") pod \"c103d769-22e3-4f04-92f6-fc9677f3098b\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " Oct 30 19:47:20.511638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511523 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-certs\") pod \"a5e31c74-47bc-453f-81ef-795c60afeb3c\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " Oct 30 19:47:20.511638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511582 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-socket\") pod \"a5e31c74-47bc-453f-81ef-795c60afeb3c\" (UID: \"a5e31c74-47bc-453f-81ef-795c60afeb3c\") " Oct 30 19:47:20.511638 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511622 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-credential-socket\") pod \"c103d769-22e3-4f04-92f6-fc9677f3098b\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " Oct 30 19:47:20.511836 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511654 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-token\" (UniqueName: \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-token\") pod \"c103d769-22e3-4f04-92f6-fc9677f3098b\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " Oct 30 19:47:20.511836 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.511702 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-podinfo\") pod \"c103d769-22e3-4f04-92f6-fc9677f3098b\" (UID: \"c103d769-22e3-4f04-92f6-fc9677f3098b\") " Oct 30 19:47:20.513654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.512163 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-data" pod "a5e31c74-47bc-453f-81ef-795c60afeb3c" (UID: "a5e31c74-47bc-453f-81ef-795c60afeb3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.513654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.512321 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.513654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.512663 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e31c74-47bc-453f-81ef-795c60afeb3c-istiod-ca-cert" pod "a5e31c74-47bc-453f-81ef-795c60afeb3c" (UID: "a5e31c74-47bc-453f-81ef-795c60afeb3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:20.513654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.512915 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-credential-socket" pod "a5e31c74-47bc-453f-81ef-795c60afeb3c" (UID: "a5e31c74-47bc-453f-81ef-795c60afeb3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.513654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.512943 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-socket" pod "a5e31c74-47bc-453f-81ef-795c60afeb3c" (UID: "a5e31c74-47bc-453f-81ef-795c60afeb3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.513654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.513156 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-socket" pod "c103d769-22e3-4f04-92f6-fc9677f3098b" (UID: "c103d769-22e3-4f04-92f6-fc9677f3098b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.513654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.513235 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-data" pod "c103d769-22e3-4f04-92f6-fc9677f3098b" (UID: "c103d769-22e3-4f04-92f6-fc9677f3098b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.513654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.513256 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c103d769-22e3-4f04-92f6-fc9677f3098b-istiod-ca-cert" pod "c103d769-22e3-4f04-92f6-fc9677f3098b" (UID: "c103d769-22e3-4f04-92f6-fc9677f3098b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:20.513654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.513347 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-certs" pod "a5e31c74-47bc-453f-81ef-795c60afeb3c" (UID: "a5e31c74-47bc-453f-81ef-795c60afeb3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.513654 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.513427 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-credential-socket" pod "c103d769-22e3-4f04-92f6-fc9677f3098b" (UID: "c103d769-22e3-4f04-92f6-fc9677f3098b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.515511 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.515473 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-kube-api-access-dk4x6" pod "c103d769-22e3-4f04-92f6-fc9677f3098b" (UID: "c103d769-22e3-4f04-92f6-fc9677f3098b"). InnerVolumeSpecName "kube-api-access-dk4x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:20.515751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.515641 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-envoy" pod "a5e31c74-47bc-453f-81ef-795c60afeb3c" (UID: "a5e31c74-47bc-453f-81ef-795c60afeb3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.515751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.515716 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-envoy" pod "c103d769-22e3-4f04-92f6-fc9677f3098b" (UID: "c103d769-22e3-4f04-92f6-fc9677f3098b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.515751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.515735 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-podinfo" pod "c103d769-22e3-4f04-92f6-fc9677f3098b" (UID: "c103d769-22e3-4f04-92f6-fc9677f3098b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:47:20.516241 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.516218 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-certs" pod "c103d769-22e3-4f04-92f6-fc9677f3098b" (UID: "c103d769-22e3-4f04-92f6-fc9677f3098b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:20.516241 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.516235 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-token" pod "c103d769-22e3-4f04-92f6-fc9677f3098b" (UID: "c103d769-22e3-4f04-92f6-fc9677f3098b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:20.516735 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.516712 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-token" pod "a5e31c74-47bc-453f-81ef-795c60afeb3c" (UID: "a5e31c74-47bc-453f-81ef-795c60afeb3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:20.517568 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.517546 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-podinfo" pod "a5e31c74-47bc-453f-81ef-795c60afeb3c" (UID: "a5e31c74-47bc-453f-81ef-795c60afeb3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:47:20.518991 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.518965 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-kube-api-access-c8m2r" pod "a5e31c74-47bc-453f-81ef-795c60afeb3c" (UID: "a5e31c74-47bc-453f-81ef-795c60afeb3c"). InnerVolumeSpecName "kube-api-access-c8m2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:20.613237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613182 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613202 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613213 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613221 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613228 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613237 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613237 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c103d769-22e3-4f04-92f6-fc9677f3098b-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613245 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e31c74-47bc-453f-81ef-795c60afeb3c-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613253 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613260 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613269 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e31c74-47bc-453f-81ef-795c60afeb3c-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613276 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613284 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613292 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613299 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613306 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk4x6\" (UniqueName: \"kubernetes.io/projected/c103d769-22e3-4f04-92f6-fc9677f3098b-kube-api-access-dk4x6\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613313 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8m2r\" (UniqueName: \"kubernetes.io/projected/a5e31c74-47bc-453f-81ef-795c60afeb3c-kube-api-access-c8m2r\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:20.613534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:20.613322 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c103d769-22e3-4f04-92f6-fc9677f3098b-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:21.055343 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.055308 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj"] Oct 30 19:47:21.055622 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.055573 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" podUID="5eec7931-fe25-49a1-a11a-95040bbf7fba" containerName="maas-api" containerID="cri-o://b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2" gracePeriod=30 Oct 30 19:47:21.162286 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.162254 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" Oct 30 19:47:21.162286 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.162269 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49" event={"ID":"08c3a4f0-8862-43a6-89ac-3e6c847a6ab2","Type":"ContainerDied","Data":"05120fdac870836dd4d711f82d611df3ec3356f416581fc2525e5baf535e99f6"} Oct 30 19:47:21.162509 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.162358 4904 scope.go:122] "RemoveContainer" containerID="0e181c0b871d188c29da9e93f01d0543d5aa0bf9869dbde5dd4413b69ef30ee3" Oct 30 19:47:21.169069 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.168862 4904 generic.go:356] "Generic (PLEG): container finished" podID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerID="28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da" exitCode=0 Oct 30 19:47:21.169388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.169363 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" event={"ID":"c103d769-22e3-4f04-92f6-fc9677f3098b","Type":"ContainerDied","Data":"28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da"} Oct 30 19:47:21.169483 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.169464 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" event={"ID":"c103d769-22e3-4f04-92f6-fc9677f3098b","Type":"ContainerDied","Data":"28deea3529374d7681f0bf5f6b5ee3fb354e431bb6e5f3f978a2e830fb2f998f"} Oct 30 19:47:21.169819 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.169796 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x" Oct 30 19:47:21.176574 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.174946 4904 generic.go:356] "Generic (PLEG): container finished" podID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerID="e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c" exitCode=0 Oct 30 19:47:21.176574 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.175014 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" event={"ID":"a5e31c74-47bc-453f-81ef-795c60afeb3c","Type":"ContainerDied","Data":"e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c"} Oct 30 19:47:21.176574 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.175065 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" event={"ID":"a5e31c74-47bc-453f-81ef-795c60afeb3c","Type":"ContainerDied","Data":"31ccaee3e6b1de528ae6976bae743bc86c1ebeade3b9cbdb15e007db552bc8dd"} Oct 30 19:47:21.176574 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.175215 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq" Oct 30 19:47:21.186852 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.186828 4904 scope.go:122] "RemoveContainer" containerID="28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da" Oct 30 19:47:21.253812 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.253446 4904 scope.go:122] "RemoveContainer" containerID="28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da" Oct 30 19:47:21.256415 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:47:21.254055 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da\": container with ID starting with 28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da not found: ID does not exist" containerID="28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da" Oct 30 19:47:21.256415 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.254118 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da"} err="failed to get container status \"28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da\": rpc error: code = NotFound desc = could not find container \"28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da\": container with ID starting with 28fe2ee084686864b14a3d951e970a0afffe6c17f68319364db9dd7fda9527da not found: ID does not exist" Oct 30 19:47:21.256415 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.254145 4904 scope.go:122] "RemoveContainer" containerID="e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c" Oct 30 19:47:21.300721 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.300658 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x"] Oct 30 19:47:21.317692 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.309251 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" podUID="5eec7931-fe25-49a1-a11a-95040bbf7fba" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.113:8443/health\": dial tcp 10.132.0.113:8443: connect: connection refused" Oct 30 19:47:21.331151 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.324996 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-zkx7x"] Oct 30 19:47:21.353296 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.350311 4904 scope.go:122] "RemoveContainer" containerID="e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c" Oct 30 19:47:21.365122 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:47:21.363393 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c\": container with ID starting with e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c not found: ID does not exist" containerID="e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c" Oct 30 19:47:21.365122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.363436 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c"} err="failed to get container status \"e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c\": rpc error: code = NotFound desc = could not find container \"e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c\": container with ID starting with e0583555250f65bc8bd8d164a5c374b149b1d04de52199339ee41d7b63b1f97c not found: ID does not exist" Oct 30 19:47:21.365122 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.364204 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49"] Oct 30 19:47:21.370410 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.370382 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-klf49"] Oct 30 19:47:21.456410 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.456370 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq"] Oct 30 19:47:21.457515 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.457404 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-x7zqq"] Oct 30 19:47:21.537682 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.537615 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:47:21.545993 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.545966 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eec7931-fe25-49a1-a11a-95040bbf7fba-kube-api-access-26zfw\" (UniqueName: \"kubernetes.io/projected/5eec7931-fe25-49a1-a11a-95040bbf7fba-kube-api-access-26zfw\") pod \"5eec7931-fe25-49a1-a11a-95040bbf7fba\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " Oct 30 19:47:21.546172 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.546018 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-metrics-certs\") pod \"5eec7931-fe25-49a1-a11a-95040bbf7fba\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " Oct 30 19:47:21.546172 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.546148 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-maas-api-tls\") pod \"5eec7931-fe25-49a1-a11a-95040bbf7fba\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " Oct 30 19:47:21.546293 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.546179 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eec7931-fe25-49a1-a11a-95040bbf7fba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eec7931-fe25-49a1-a11a-95040bbf7fba-openshift-service-ca\") pod \"5eec7931-fe25-49a1-a11a-95040bbf7fba\" (UID: \"5eec7931-fe25-49a1-a11a-95040bbf7fba\") " Oct 30 19:47:21.546607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.546576 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eec7931-fe25-49a1-a11a-95040bbf7fba-openshift-service-ca" pod "5eec7931-fe25-49a1-a11a-95040bbf7fba" (UID: "5eec7931-fe25-49a1-a11a-95040bbf7fba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:21.546741 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.546727 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eec7931-fe25-49a1-a11a-95040bbf7fba-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:21.548814 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.548789 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-maas-api-tls" pod "5eec7931-fe25-49a1-a11a-95040bbf7fba" (UID: "5eec7931-fe25-49a1-a11a-95040bbf7fba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:21.548972 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.548952 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-metrics-certs" pod "5eec7931-fe25-49a1-a11a-95040bbf7fba" (UID: "5eec7931-fe25-49a1-a11a-95040bbf7fba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:21.549049 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.549034 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eec7931-fe25-49a1-a11a-95040bbf7fba-kube-api-access-26zfw" pod "5eec7931-fe25-49a1-a11a-95040bbf7fba" (UID: "5eec7931-fe25-49a1-a11a-95040bbf7fba"). InnerVolumeSpecName "kube-api-access-26zfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:21.648536 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.648503 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:21.648536 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.648539 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26zfw\" (UniqueName: \"kubernetes.io/projected/5eec7931-fe25-49a1-a11a-95040bbf7fba-kube-api-access-26zfw\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:21.648733 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:21.648555 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eec7931-fe25-49a1-a11a-95040bbf7fba-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:22.185690 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:22.185634 4904 generic.go:356] "Generic (PLEG): container finished" podID="5eec7931-fe25-49a1-a11a-95040bbf7fba" containerID="b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2" exitCode=0 Oct 30 19:47:22.185886 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:22.185703 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" event={"ID":"5eec7931-fe25-49a1-a11a-95040bbf7fba","Type":"ContainerDied","Data":"b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2"} Oct 30 19:47:22.185886 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:22.185786 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" event={"ID":"5eec7931-fe25-49a1-a11a-95040bbf7fba","Type":"ContainerDied","Data":"1c8f3488e06a106d6eab06d292cc561cad770dd04e3f2c17682079042271edb6"} Oct 30 19:47:22.185886 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:22.185827 4904 scope.go:122] "RemoveContainer" containerID="b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2" Oct 30 19:47:22.187142 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:22.186097 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj" Oct 30 19:47:22.197252 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:22.197234 4904 scope.go:122] "RemoveContainer" containerID="b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2" Oct 30 19:47:22.197535 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:47:22.197507 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2\": container with ID starting with b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2 not found: ID does not exist" containerID="b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2" Oct 30 19:47:22.197637 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:22.197542 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2"} err="failed to get container status \"b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2\": rpc error: code = NotFound desc = could not find container \"b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2\": container with ID starting with b8496e868552c080ee628760acf3e459a7bb5707cb3a9850141b80ccebfa6fb2 not found: ID does not exist" Oct 30 19:47:22.246635 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:22.246611 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj"] Oct 30 19:47:22.254543 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:22.254521 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-291d35-6c55bb6f76-gjngj"] Oct 30 19:47:23.050595 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.050574 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:47:23.066621 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.066600 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-podinfo\") pod \"08afa540-378d-4cc8-8a7c-7173ddffb742\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " Oct 30 19:47:23.066737 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.066643 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08afa540-378d-4cc8-8a7c-7173ddffb742-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08afa540-378d-4cc8-8a7c-7173ddffb742-istiod-ca-cert\") pod \"08afa540-378d-4cc8-8a7c-7173ddffb742\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " Oct 30 19:47:23.066737 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.066702 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-kube-api-access-jl7wr\" (UniqueName: \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-kube-api-access-jl7wr\") pod \"08afa540-378d-4cc8-8a7c-7173ddffb742\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " Oct 30 19:47:23.066815 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.066735 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-data\") pod \"08afa540-378d-4cc8-8a7c-7173ddffb742\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " Oct 30 19:47:23.066815 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.066774 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-credential-socket\") pod \"08afa540-378d-4cc8-8a7c-7173ddffb742\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " Oct 30 19:47:23.066911 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.066889 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-socket\") pod \"08afa540-378d-4cc8-8a7c-7173ddffb742\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " Oct 30 19:47:23.066980 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.066924 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-envoy\") pod \"08afa540-378d-4cc8-8a7c-7173ddffb742\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " Oct 30 19:47:23.067041 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.066996 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-token\" (UniqueName: \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-token\") pod \"08afa540-378d-4cc8-8a7c-7173ddffb742\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " Oct 30 19:47:23.067041 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067038 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-certs\") pod \"08afa540-378d-4cc8-8a7c-7173ddffb742\" (UID: \"08afa540-378d-4cc8-8a7c-7173ddffb742\") " Oct 30 19:47:23.067182 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067125 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08afa540-378d-4cc8-8a7c-7173ddffb742-istiod-ca-cert" pod "08afa540-378d-4cc8-8a7c-7173ddffb742" (UID: "08afa540-378d-4cc8-8a7c-7173ddffb742"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:23.067246 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067206 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-credential-socket" pod "08afa540-378d-4cc8-8a7c-7173ddffb742" (UID: "08afa540-378d-4cc8-8a7c-7173ddffb742"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:23.067341 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067250 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-socket" pod "08afa540-378d-4cc8-8a7c-7173ddffb742" (UID: "08afa540-378d-4cc8-8a7c-7173ddffb742"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:23.067341 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067316 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-data" pod "08afa540-378d-4cc8-8a7c-7173ddffb742" (UID: "08afa540-378d-4cc8-8a7c-7173ddffb742"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:23.067552 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067507 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-certs" pod "08afa540-378d-4cc8-8a7c-7173ddffb742" (UID: "08afa540-378d-4cc8-8a7c-7173ddffb742"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:23.067679 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067665 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:23.067743 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067682 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08afa540-378d-4cc8-8a7c-7173ddffb742-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:23.067743 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067690 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:23.067743 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067698 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:23.067743 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.067706 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:23.069038 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.069015 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-podinfo" pod "08afa540-378d-4cc8-8a7c-7173ddffb742" (UID: "08afa540-378d-4cc8-8a7c-7173ddffb742"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:47:23.069542 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.069519 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-token" pod "08afa540-378d-4cc8-8a7c-7173ddffb742" (UID: "08afa540-378d-4cc8-8a7c-7173ddffb742"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:23.069797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.069775 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-envoy" pod "08afa540-378d-4cc8-8a7c-7173ddffb742" (UID: "08afa540-378d-4cc8-8a7c-7173ddffb742"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:23.069797 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.069786 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-kube-api-access-jl7wr" pod "08afa540-378d-4cc8-8a7c-7173ddffb742" (UID: "08afa540-378d-4cc8-8a7c-7173ddffb742"). InnerVolumeSpecName "kube-api-access-jl7wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:23.169028 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.168995 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:23.169028 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.169029 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:23.169028 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.169042 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08afa540-378d-4cc8-8a7c-7173ddffb742-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:23.169267 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.169056 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7wr\" (UniqueName: \"kubernetes.io/projected/08afa540-378d-4cc8-8a7c-7173ddffb742-kube-api-access-jl7wr\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:23.197716 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.197684 4904 generic.go:356] "Generic (PLEG): container finished" podID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerID="ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651" exitCode=0 Oct 30 19:47:23.197872 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.197733 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" Oct 30 19:47:23.197872 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.197768 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" event={"ID":"08afa540-378d-4cc8-8a7c-7173ddffb742","Type":"ContainerDied","Data":"ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651"} Oct 30 19:47:23.197872 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.197821 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z" event={"ID":"08afa540-378d-4cc8-8a7c-7173ddffb742","Type":"ContainerDied","Data":"5be8d397723f7908d87143c27702c788ee8c7cd63f0128326ed457f96f3020ad"} Oct 30 19:47:23.197872 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.197851 4904 scope.go:122] "RemoveContainer" containerID="ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651" Oct 30 19:47:23.217143 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.216322 4904 scope.go:122] "RemoveContainer" containerID="ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651" Oct 30 19:47:23.221619 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:47:23.221584 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651\": container with ID starting with ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651 not found: ID does not exist" containerID="ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651" Oct 30 19:47:23.221752 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.221627 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651"} err="failed to get container status \"ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651\": rpc error: code = NotFound desc = could not find container \"ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651\": container with ID starting with ac83e9d1d4205a010fa85ecb3a4b88c2d5d19bd3c15bbaa2726e89443bd38651 not found: ID does not exist" Oct 30 19:47:23.227499 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.227476 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" path="/var/lib/kubelet/pods/08c3a4f0-8862-43a6-89ac-3e6c847a6ab2/volumes" Oct 30 19:47:23.228009 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.227989 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eec7931-fe25-49a1-a11a-95040bbf7fba" path="/var/lib/kubelet/pods/5eec7931-fe25-49a1-a11a-95040bbf7fba/volumes" Oct 30 19:47:23.228549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.228530 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" path="/var/lib/kubelet/pods/a5e31c74-47bc-453f-81ef-795c60afeb3c/volumes" Oct 30 19:47:23.229341 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.229323 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" path="/var/lib/kubelet/pods/c103d769-22e3-4f04-92f6-fc9677f3098b/volumes" Oct 30 19:47:23.277444 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.277381 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z"] Oct 30 19:47:23.296977 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:23.296956 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-psk4z"] Oct 30 19:47:24.511140 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:47:24.511087 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:47:25.222667 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:25.222633 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" path="/var/lib/kubelet/pods/08afa540-378d-4cc8-8a7c-7173ddffb742/volumes" Oct 30 19:47:34.286813 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.286743 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" containerID="cri-o://08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8" gracePeriod=45 Oct 30 19:47:34.302267 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.301990 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" containerID="cri-o://98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d" gracePeriod=45 Oct 30 19:47:34.657121 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.654815 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:47:34.749179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.746904 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:47:34.765588 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.765556 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-dshm\") pod \"b26d1d6e-fb62-4e3c-9913-2944866d3508\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " Oct 30 19:47:34.774244 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.774219 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26d1d6e-fb62-4e3c-9913-2944866d3508-tls-certs\" (UniqueName: \"kubernetes.io/secret/b26d1d6e-fb62-4e3c-9913-2944866d3508-tls-certs\") pod \"b26d1d6e-fb62-4e3c-9913-2944866d3508\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " Oct 30 19:47:34.774820 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.774800 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26d1d6e-fb62-4e3c-9913-2944866d3508-kube-api-access-qgmh5\" (UniqueName: \"kubernetes.io/projected/b26d1d6e-fb62-4e3c-9913-2944866d3508-kube-api-access-qgmh5\") pod \"b26d1d6e-fb62-4e3c-9913-2944866d3508\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " Oct 30 19:47:34.775029 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.775002 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-model-cache\") pod \"b26d1d6e-fb62-4e3c-9913-2944866d3508\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " Oct 30 19:47:34.775476 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.775339 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-model-cache" pod "b26d1d6e-fb62-4e3c-9913-2944866d3508" (UID: "b26d1d6e-fb62-4e3c-9913-2944866d3508"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:34.776668 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.775638 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-home\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-home\") pod \"b26d1d6e-fb62-4e3c-9913-2944866d3508\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " Oct 30 19:47:34.776668 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.775700 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-tmp-dir\") pod \"b26d1d6e-fb62-4e3c-9913-2944866d3508\" (UID: \"b26d1d6e-fb62-4e3c-9913-2944866d3508\") " Oct 30 19:47:34.776668 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.776237 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.776668 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.776583 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-home" pod "b26d1d6e-fb62-4e3c-9913-2944866d3508" (UID: "b26d1d6e-fb62-4e3c-9913-2944866d3508"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:34.776668 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.776634 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-tmp-dir" pod "b26d1d6e-fb62-4e3c-9913-2944866d3508" (UID: "b26d1d6e-fb62-4e3c-9913-2944866d3508"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:34.784436 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.778521 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-dshm" pod "b26d1d6e-fb62-4e3c-9913-2944866d3508" (UID: "b26d1d6e-fb62-4e3c-9913-2944866d3508"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:34.784436 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.778594 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26d1d6e-fb62-4e3c-9913-2944866d3508-tls-certs" pod "b26d1d6e-fb62-4e3c-9913-2944866d3508" (UID: "b26d1d6e-fb62-4e3c-9913-2944866d3508"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:34.805152 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.800189 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26d1d6e-fb62-4e3c-9913-2944866d3508-kube-api-access-qgmh5" pod "b26d1d6e-fb62-4e3c-9913-2944866d3508" (UID: "b26d1d6e-fb62-4e3c-9913-2944866d3508"). InnerVolumeSpecName "kube-api-access-qgmh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.876799 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-model-cache\") pod \"e51692e8-fa0c-4042-8255-be65097e79a9\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.876864 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51692e8-fa0c-4042-8255-be65097e79a9-kube-api-access-5fgrr\" (UniqueName: \"kubernetes.io/projected/e51692e8-fa0c-4042-8255-be65097e79a9-kube-api-access-5fgrr\") pod \"e51692e8-fa0c-4042-8255-be65097e79a9\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.876985 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-dshm\") pod \"e51692e8-fa0c-4042-8255-be65097e79a9\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.877047 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-tmp-dir\") pod \"e51692e8-fa0c-4042-8255-be65097e79a9\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.877207 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e51692e8-fa0c-4042-8255-be65097e79a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/e51692e8-fa0c-4042-8255-be65097e79a9-tls-certs\") pod \"e51692e8-fa0c-4042-8255-be65097e79a9\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.877306 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-home\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-home\") pod \"e51692e8-fa0c-4042-8255-be65097e79a9\" (UID: \"e51692e8-fa0c-4042-8255-be65097e79a9\") " Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.877043 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-model-cache" pod "e51692e8-fa0c-4042-8255-be65097e79a9" (UID: "e51692e8-fa0c-4042-8255-be65097e79a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.877462 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-tmp-dir" pod "e51692e8-fa0c-4042-8255-be65097e79a9" (UID: "e51692e8-fa0c-4042-8255-be65097e79a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.878086 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.878140 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.878156 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.878168 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b26d1d6e-fb62-4e3c-9913-2944866d3508-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.878179 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.878209 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b26d1d6e-fb62-4e3c-9913-2944866d3508-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.878224 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgmh5\" (UniqueName: \"kubernetes.io/projected/b26d1d6e-fb62-4e3c-9913-2944866d3508-kube-api-access-qgmh5\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.879123 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.878087 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-home" pod "e51692e8-fa0c-4042-8255-be65097e79a9" (UID: "e51692e8-fa0c-4042-8255-be65097e79a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:34.880713 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.880683 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51692e8-fa0c-4042-8255-be65097e79a9-tls-certs" pod "e51692e8-fa0c-4042-8255-be65097e79a9" (UID: "e51692e8-fa0c-4042-8255-be65097e79a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:34.888152 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.883286 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-dshm" pod "e51692e8-fa0c-4042-8255-be65097e79a9" (UID: "e51692e8-fa0c-4042-8255-be65097e79a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:34.888152 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.883761 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51692e8-fa0c-4042-8255-be65097e79a9-kube-api-access-5fgrr" pod "e51692e8-fa0c-4042-8255-be65097e79a9" (UID: "e51692e8-fa0c-4042-8255-be65097e79a9"). InnerVolumeSpecName "kube-api-access-5fgrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:34.979549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.979500 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.979549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.979534 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51692e8-fa0c-4042-8255-be65097e79a9-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.979549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.979545 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51692e8-fa0c-4042-8255-be65097e79a9-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:34.979549 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:34.979560 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgrr\" (UniqueName: \"kubernetes.io/projected/e51692e8-fa0c-4042-8255-be65097e79a9-kube-api-access-5fgrr\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:35.339128 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.339043 4904 generic.go:356] "Generic (PLEG): container finished" podID="e51692e8-fa0c-4042-8255-be65097e79a9" containerID="98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d" exitCode=0 Oct 30 19:47:35.339593 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.339369 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" event={"ID":"e51692e8-fa0c-4042-8255-be65097e79a9","Type":"ContainerDied","Data":"98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d"} Oct 30 19:47:35.339593 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.339409 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" event={"ID":"e51692e8-fa0c-4042-8255-be65097e79a9","Type":"ContainerDied","Data":"1255cc6397423d014fcf3d846275236a3b9448e2c82433d35fad0232d511c4c6"} Oct 30 19:47:35.339593 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.339448 4904 scope.go:122] "RemoveContainer" containerID="98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d" Oct 30 19:47:35.339772 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.339626 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" Oct 30 19:47:35.352134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.350607 4904 generic.go:356] "Generic (PLEG): container finished" podID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerID="08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8" exitCode=0 Oct 30 19:47:35.352134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.350677 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" event={"ID":"b26d1d6e-fb62-4e3c-9913-2944866d3508","Type":"ContainerDied","Data":"08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8"} Oct 30 19:47:35.352134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.350712 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" event={"ID":"b26d1d6e-fb62-4e3c-9913-2944866d3508","Type":"ContainerDied","Data":"5ff0603c05bc857ff1143f59bd29ccb288241351e013e7d658b67ab60a0a9814"} Oct 30 19:47:35.352134 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.350799 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2" Oct 30 19:47:35.383316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.380780 4904 scope.go:122] "RemoveContainer" containerID="98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d" Oct 30 19:47:35.383316 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:47:35.381575 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d\": container with ID starting with 98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d not found: ID does not exist" containerID="98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d" Oct 30 19:47:35.383316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.381630 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d"} err="failed to get container status \"98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d\": rpc error: code = NotFound desc = could not find container \"98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d\": container with ID starting with 98fc59547e0e01a2eff424e8d19816f52b1eb822dab9f9c52cb5f510e94a7a8d not found: ID does not exist" Oct 30 19:47:35.383316 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.381654 4904 scope.go:122] "RemoveContainer" containerID="08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8" Oct 30 19:47:35.438126 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.428170 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2"] Oct 30 19:47:35.465453 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.465404 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/premium-w4-f913c6-kserve-96d7bd8b6-7cwj2"] Oct 30 19:47:35.492286 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.487158 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5"] Oct 30 19:47:35.492522 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.492501 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5"] Oct 30 19:47:35.506716 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.506693 4904 scope.go:122] "RemoveContainer" containerID="08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8" Oct 30 19:47:35.507698 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:47:35.507673 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8\": container with ID starting with 08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8 not found: ID does not exist" containerID="08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8" Oct 30 19:47:35.507849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.507818 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8"} err="failed to get container status \"08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8\": rpc error: code = NotFound desc = could not find container \"08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8\": container with ID starting with 08ca979a94bd0807c5a0d119d5adb4bc17ae986ca08f1ca521f2fc4ade4468a8 not found: ID does not exist" Oct 30 19:47:35.678218 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:35.678178 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/simulator-w4-f913c6-kserve-5fd98c474c-dhmw5" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:47:36.179474 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:36.179444 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj"] Oct 30 19:47:36.201891 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:36.201862 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m"] Oct 30 19:47:37.258758 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:37.258722 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" path="/var/lib/kubelet/pods/b26d1d6e-fb62-4e3c-9913-2944866d3508/volumes" Oct 30 19:47:37.260028 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:37.260005 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" path="/var/lib/kubelet/pods/e51692e8-fa0c-4042-8255-be65097e79a9/volumes" Oct 30 19:47:45.753341 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:45.753310 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj"] Oct 30 19:47:45.753777 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:45.753534 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" podUID="fd683264-cd99-440c-b137-4c8b454b492f" containerName="maas-api" containerID="cri-o://1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775" gracePeriod=30 Oct 30 19:47:46.186740 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.186713 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:47:46.272070 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.272039 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd683264-cd99-440c-b137-4c8b454b492f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd683264-cd99-440c-b137-4c8b454b492f-openshift-service-ca\") pod \"fd683264-cd99-440c-b137-4c8b454b492f\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " Oct 30 19:47:46.272301 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.272172 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-maas-api-tls\") pod \"fd683264-cd99-440c-b137-4c8b454b492f\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " Oct 30 19:47:46.272301 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.272215 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-metrics-certs\") pod \"fd683264-cd99-440c-b137-4c8b454b492f\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " Oct 30 19:47:46.272571 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.272343 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd683264-cd99-440c-b137-4c8b454b492f-kube-api-access-7nnnx\" (UniqueName: \"kubernetes.io/projected/fd683264-cd99-440c-b137-4c8b454b492f-kube-api-access-7nnnx\") pod \"fd683264-cd99-440c-b137-4c8b454b492f\" (UID: \"fd683264-cd99-440c-b137-4c8b454b492f\") " Oct 30 19:47:46.272571 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.272459 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd683264-cd99-440c-b137-4c8b454b492f-openshift-service-ca" pod "fd683264-cd99-440c-b137-4c8b454b492f" (UID: "fd683264-cd99-440c-b137-4c8b454b492f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:46.272806 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.272783 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd683264-cd99-440c-b137-4c8b454b492f-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:46.275534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.275509 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-metrics-certs" pod "fd683264-cd99-440c-b137-4c8b454b492f" (UID: "fd683264-cd99-440c-b137-4c8b454b492f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:46.275534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.275531 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd683264-cd99-440c-b137-4c8b454b492f-kube-api-access-7nnnx" pod "fd683264-cd99-440c-b137-4c8b454b492f" (UID: "fd683264-cd99-440c-b137-4c8b454b492f"). InnerVolumeSpecName "kube-api-access-7nnnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:46.275716 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.275615 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-maas-api-tls" pod "fd683264-cd99-440c-b137-4c8b454b492f" (UID: "fd683264-cd99-440c-b137-4c8b454b492f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:46.374086 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.374058 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:46.374086 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.374084 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd683264-cd99-440c-b137-4c8b454b492f-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:46.374086 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.374094 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nnnx\" (UniqueName: \"kubernetes.io/projected/fd683264-cd99-440c-b137-4c8b454b492f-kube-api-access-7nnnx\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:46.475216 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.475181 4904 generic.go:356] "Generic (PLEG): container finished" podID="fd683264-cd99-440c-b137-4c8b454b492f" containerID="1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775" exitCode=0 Oct 30 19:47:46.475375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.475239 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" Oct 30 19:47:46.475375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.475270 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" event={"ID":"fd683264-cd99-440c-b137-4c8b454b492f","Type":"ContainerDied","Data":"1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775"} Oct 30 19:47:46.475375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.475338 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj" event={"ID":"fd683264-cd99-440c-b137-4c8b454b492f","Type":"ContainerDied","Data":"4a0f8b88337aafbf43d489c5b5e18dd44a91f6632784b2dc7a54f5c7545bafec"} Oct 30 19:47:46.475375 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.475367 4904 scope.go:122] "RemoveContainer" containerID="1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775" Oct 30 19:47:46.486363 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.486340 4904 scope.go:122] "RemoveContainer" containerID="1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775" Oct 30 19:47:46.486658 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:47:46.486636 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775\": container with ID starting with 1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775 not found: ID does not exist" containerID="1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775" Oct 30 19:47:46.486735 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.486673 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775"} err="failed to get container status \"1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775\": rpc error: code = NotFound desc = could not find container \"1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775\": container with ID starting with 1d6bf0c7018fe5128f6c06fb8f12a482f1bcbe786e323b395edaa9a372db1775 not found: ID does not exist" Oct 30 19:47:46.536361 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.536328 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj"] Oct 30 19:47:46.540241 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:46.540218 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f913c6-fd868c5b9-rdhfj"] Oct 30 19:47:47.229119 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:47.229067 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd683264-cd99-440c-b137-4c8b454b492f" path="/var/lib/kubelet/pods/fd683264-cd99-440c-b137-4c8b454b492f/volumes" Oct 30 19:47:51.330276 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.330220 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" containerID="cri-o://57965e0a4d8195629dee879f79f939a8e73f7b50dab57ed34258f6a52c387c4a" gracePeriod=45 Oct 30 19:47:51.394006 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.393978 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" containerID="cri-o://daee3efb808b85d571fe347857d2244eac9b9f5336982bcc6c160424fcd0ec56" gracePeriod=45 Oct 30 19:47:51.514274 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.514239 4904 generic.go:356] "Generic (PLEG): container finished" podID="86e70905-a057-43ca-864c-52a11083e726" containerID="daee3efb808b85d571fe347857d2244eac9b9f5336982bcc6c160424fcd0ec56" exitCode=0 Oct 30 19:47:51.514386 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.514359 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" event={"ID":"86e70905-a057-43ca-864c-52a11083e726","Type":"ContainerDied","Data":"daee3efb808b85d571fe347857d2244eac9b9f5336982bcc6c160424fcd0ec56"} Oct 30 19:47:51.516833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.516675 4904 generic.go:356] "Generic (PLEG): container finished" podID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerID="57965e0a4d8195629dee879f79f939a8e73f7b50dab57ed34258f6a52c387c4a" exitCode=0 Oct 30 19:47:51.516833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.516725 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" event={"ID":"a8ea566e-8d44-469a-bd0b-2b2932b16679","Type":"ContainerDied","Data":"57965e0a4d8195629dee879f79f939a8e73f7b50dab57ed34258f6a52c387c4a"} Oct 30 19:47:51.770351 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.770325 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:47:51.774362 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.774339 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.834937 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-home\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-home\") pod \"86e70905-a057-43ca-864c-52a11083e726\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835081 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-tmp-dir\") pod \"86e70905-a057-43ca-864c-52a11083e726\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835145 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-model-cache\") pod \"86e70905-a057-43ca-864c-52a11083e726\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835201 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-dshm\") pod \"86e70905-a057-43ca-864c-52a11083e726\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835253 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-dshm\") pod \"a8ea566e-8d44-469a-bd0b-2b2932b16679\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835367 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-model-cache\") pod \"a8ea566e-8d44-469a-bd0b-2b2932b16679\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835420 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ea566e-8d44-469a-bd0b-2b2932b16679-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8ea566e-8d44-469a-bd0b-2b2932b16679-tls-certs\") pod \"a8ea566e-8d44-469a-bd0b-2b2932b16679\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835502 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86e70905-a057-43ca-864c-52a11083e726-tls-certs\" (UniqueName: \"kubernetes.io/secret/86e70905-a057-43ca-864c-52a11083e726-tls-certs\") pod \"86e70905-a057-43ca-864c-52a11083e726\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835536 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e70905-a057-43ca-864c-52a11083e726-kube-api-access-2fgkm\" (UniqueName: \"kubernetes.io/projected/86e70905-a057-43ca-864c-52a11083e726-kube-api-access-2fgkm\") pod \"86e70905-a057-43ca-864c-52a11083e726\" (UID: \"86e70905-a057-43ca-864c-52a11083e726\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835666 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ea566e-8d44-469a-bd0b-2b2932b16679-kube-api-access-rg4k2\" (UniqueName: \"kubernetes.io/projected/a8ea566e-8d44-469a-bd0b-2b2932b16679-kube-api-access-rg4k2\") pod \"a8ea566e-8d44-469a-bd0b-2b2932b16679\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835719 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-tmp-dir\") pod \"a8ea566e-8d44-469a-bd0b-2b2932b16679\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.835752 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-home\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-home\") pod \"a8ea566e-8d44-469a-bd0b-2b2932b16679\" (UID: \"a8ea566e-8d44-469a-bd0b-2b2932b16679\") " Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.836320 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-home" pod "a8ea566e-8d44-469a-bd0b-2b2932b16679" (UID: "a8ea566e-8d44-469a-bd0b-2b2932b16679"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.836480 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-home" pod "86e70905-a057-43ca-864c-52a11083e726" (UID: "86e70905-a057-43ca-864c-52a11083e726"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.836608 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-tmp-dir" pod "86e70905-a057-43ca-864c-52a11083e726" (UID: "86e70905-a057-43ca-864c-52a11083e726"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.837171 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-model-cache" pod "86e70905-a057-43ca-864c-52a11083e726" (UID: "86e70905-a057-43ca-864c-52a11083e726"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.838135 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-model-cache" pod "a8ea566e-8d44-469a-bd0b-2b2932b16679" (UID: "a8ea566e-8d44-469a-bd0b-2b2932b16679"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:51.839168 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.838268 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-tmp-dir" pod "a8ea566e-8d44-469a-bd0b-2b2932b16679" (UID: "a8ea566e-8d44-469a-bd0b-2b2932b16679"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:51.847656 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.847575 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-dshm" pod "a8ea566e-8d44-469a-bd0b-2b2932b16679" (UID: "a8ea566e-8d44-469a-bd0b-2b2932b16679"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:51.847656 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.847618 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ea566e-8d44-469a-bd0b-2b2932b16679-kube-api-access-rg4k2" pod "a8ea566e-8d44-469a-bd0b-2b2932b16679" (UID: "a8ea566e-8d44-469a-bd0b-2b2932b16679"). InnerVolumeSpecName "kube-api-access-rg4k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:51.847989 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.847962 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ea566e-8d44-469a-bd0b-2b2932b16679-tls-certs" pod "a8ea566e-8d44-469a-bd0b-2b2932b16679" (UID: "a8ea566e-8d44-469a-bd0b-2b2932b16679"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:51.849898 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.849843 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-dshm" pod "86e70905-a057-43ca-864c-52a11083e726" (UID: "86e70905-a057-43ca-864c-52a11083e726"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:51.849898 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.849852 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e70905-a057-43ca-864c-52a11083e726-kube-api-access-2fgkm" pod "86e70905-a057-43ca-864c-52a11083e726" (UID: "86e70905-a057-43ca-864c-52a11083e726"). InnerVolumeSpecName "kube-api-access-2fgkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:51.849898 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.849857 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e70905-a057-43ca-864c-52a11083e726-tls-certs" pod "86e70905-a057-43ca-864c-52a11083e726" (UID: "86e70905-a057-43ca-864c-52a11083e726"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:51.937373 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937337 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937373 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937367 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937373 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937378 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8ea566e-8d44-469a-bd0b-2b2932b16679-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937386 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e70905-a057-43ca-864c-52a11083e726-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937397 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fgkm\" (UniqueName: \"kubernetes.io/projected/86e70905-a057-43ca-864c-52a11083e726-kube-api-access-2fgkm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937406 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4k2\" (UniqueName: \"kubernetes.io/projected/a8ea566e-8d44-469a-bd0b-2b2932b16679-kube-api-access-rg4k2\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937415 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937424 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8ea566e-8d44-469a-bd0b-2b2932b16679-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937431 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937440 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937447 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:51.937671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:51.937454 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e70905-a057-43ca-864c-52a11083e726-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:47:52.528502 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:52.528468 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" Oct 30 19:47:52.528983 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:52.528503 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj" event={"ID":"86e70905-a057-43ca-864c-52a11083e726","Type":"ContainerDied","Data":"fae1b0d9b6f8f4692a02290606b02798b6ff38c0da7613682835fe6921597cc3"} Oct 30 19:47:52.528983 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:52.528553 4904 scope.go:122] "RemoveContainer" containerID="daee3efb808b85d571fe347857d2244eac9b9f5336982bcc6c160424fcd0ec56" Oct 30 19:47:52.531812 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:52.531790 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" event={"ID":"a8ea566e-8d44-469a-bd0b-2b2932b16679","Type":"ContainerDied","Data":"3d389157625ddc99b92a063c5dac43b29f4b7eb8aaf27f302fa7007fe4203d89"} Oct 30 19:47:52.531917 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:52.531865 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m" Oct 30 19:47:52.544551 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:52.544531 4904 scope.go:122] "RemoveContainer" containerID="57965e0a4d8195629dee879f79f939a8e73f7b50dab57ed34258f6a52c387c4a" Oct 30 19:47:52.604968 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:52.604898 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m"] Oct 30 19:47:52.618786 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:52.618760 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/simulator-w1-d797d2-kserve-5b86b9f589-bjn2m"] Oct 30 19:47:52.670369 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:52.670329 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj"] Oct 30 19:47:52.676804 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:52.676777 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-w1-d797d2-kserve-7bc57b885f-h96rj"] Oct 30 19:47:53.258905 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:53.258863 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e70905-a057-43ca-864c-52a11083e726" path="/var/lib/kubelet/pods/86e70905-a057-43ca-864c-52a11083e726/volumes" Oct 30 19:47:53.259496 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:53.259474 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" path="/var/lib/kubelet/pods/a8ea566e-8d44-469a-bd0b-2b2932b16679/volumes" Oct 30 19:47:54.512262 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:47:54.512229 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:47:56.630609 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:56.630573 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8"] Oct 30 19:47:56.631748 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:56.631694 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" containerID="cri-o://d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09" gracePeriod=30 Oct 30 19:47:56.642815 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:56.642787 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk"] Oct 30 19:47:56.643040 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:56.643011 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerName="istio-proxy" containerID="cri-o://d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773" gracePeriod=30 Oct 30 19:47:58.372627 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:58.372593 4904 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:47:58.373096 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:47:58.372649 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:48:01.920029 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:01.919972 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:48:01.975205 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:01.975074 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:48:02.051388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051342 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-certs\") pod \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " Oct 30 19:48:02.051388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051390 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-token\" (UniqueName: \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-token\") pod \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " Oct 30 19:48:02.051675 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051452 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-socket\") pod \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " Oct 30 19:48:02.051675 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051484 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-data\") pod \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " Oct 30 19:48:02.051675 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051518 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-envoy\") pod \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " Oct 30 19:48:02.051675 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051567 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-credential-socket\") pod \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " Oct 30 19:48:02.051675 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051625 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-certs" pod "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" (UID: "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.051675 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051633 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-socket\") pod \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " Oct 30 19:48:02.051675 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051672 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-envoy\") pod \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " Oct 30 19:48:02.052097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051722 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-podinfo\") pod \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " Oct 30 19:48:02.052097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051785 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-credential-socket\") pod \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " Oct 30 19:48:02.052097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051822 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-data" pod "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" (UID: "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.052097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051853 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istiod-ca-cert\") pod \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " Oct 30 19:48:02.052097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051893 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-kube-api-access-fnzl5\" (UniqueName: \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-kube-api-access-fnzl5\") pod \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " Oct 30 19:48:02.052097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051933 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-certs\") pod \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " Oct 30 19:48:02.052097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.051986 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-data\") pod \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " Oct 30 19:48:02.052097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.052023 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-token\" (UniqueName: \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-token\") pod \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " Oct 30 19:48:02.052097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.052063 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-podinfo\") pod \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " Oct 30 19:48:02.052691 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.052137 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-kube-api-access-kktzm\" (UniqueName: \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-kube-api-access-kktzm\") pod \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\" (UID: \"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82\") " Oct 30 19:48:02.052691 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.052217 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istiod-ca-cert\") pod \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\" (UID: \"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7\") " Oct 30 19:48:02.052691 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.052326 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-socket" pod "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" (UID: "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.052691 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.052642 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istiod-ca-cert" pod "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" (UID: "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:02.052935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.052903 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.052935 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.052929 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.053058 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.052944 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.053058 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.052960 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.053903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.053366 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istiod-ca-cert" pod "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" (UID: "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:02.054635 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.054548 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-credential-socket" pod "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" (UID: "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.055500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.055470 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-token" pod "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" (UID: "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:02.055771 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.055746 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-socket" pod "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" (UID: "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.056266 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.056014 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-certs" pod "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" (UID: "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.056543 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.056512 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-data" pod "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" (UID: "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.056767 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.056745 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-credential-socket" pod "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" (UID: "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.057255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.057228 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-envoy" pod "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" (UID: "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.057380 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.057282 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-token" pod "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" (UID: "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:02.057711 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.057682 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-kube-api-access-fnzl5" pod "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" (UID: "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7"). InnerVolumeSpecName "kube-api-access-fnzl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:02.058179 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.058143 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-podinfo" pod "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" (UID: "ac0f4d7b-c714-43f5-9bb0-58a6efb416b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:48:02.058750 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.058725 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-kube-api-access-kktzm" pod "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" (UID: "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82"). InnerVolumeSpecName "kube-api-access-kktzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:02.060112 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.060067 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-envoy" pod "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" (UID: "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.065066 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.065041 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-podinfo" pod "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" (UID: "7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:48:02.153805 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153776 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnzl5\" (UniqueName: \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-kube-api-access-fnzl5\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153805 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153804 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153813 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153821 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153828 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153836 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kktzm\" (UniqueName: \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-kube-api-access-kktzm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153844 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153851 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153858 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153866 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153873 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153881 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153888 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.153958 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.153895 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.609716 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.609679 4904 generic.go:356] "Generic (PLEG): container finished" podID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerID="d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773" exitCode=0 Oct 30 19:48:02.609903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.609722 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" event={"ID":"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82","Type":"ContainerDied","Data":"d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773"} Oct 30 19:48:02.609903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.609746 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" Oct 30 19:48:02.609903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.609787 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk" event={"ID":"7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82","Type":"ContainerDied","Data":"68729d1692b10a6f46cf31ca36b833ec7ebd5e49ea3b84d7477194ea7638bb36"} Oct 30 19:48:02.609903 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.609814 4904 scope.go:122] "RemoveContainer" containerID="d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773" Oct 30 19:48:02.618591 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.618564 4904 generic.go:356] "Generic (PLEG): container finished" podID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerID="d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09" exitCode=0 Oct 30 19:48:02.618730 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.618642 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" Oct 30 19:48:02.618730 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.618651 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" event={"ID":"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7","Type":"ContainerDied","Data":"d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09"} Oct 30 19:48:02.618730 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.618712 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8" event={"ID":"ac0f4d7b-c714-43f5-9bb0-58a6efb416b7","Type":"ContainerDied","Data":"e78618b7b9918de2184715b408b3eacca7c43e774782a09e0cfaf41ea2ae81a8"} Oct 30 19:48:02.623937 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.623921 4904 scope.go:122] "RemoveContainer" containerID="d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773" Oct 30 19:48:02.624258 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:48:02.624238 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773\": container with ID starting with d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773 not found: ID does not exist" containerID="d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773" Oct 30 19:48:02.624340 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.624265 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773"} err="failed to get container status \"d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773\": rpc error: code = NotFound desc = could not find container \"d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773\": container with ID starting with d3f45cd864e9a5ccd306b0b66e182a776a68bb43ade912ceba5499f2921a3773 not found: ID does not exist" Oct 30 19:48:02.624340 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.624283 4904 scope.go:122] "RemoveContainer" containerID="d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09" Oct 30 19:48:02.633602 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.633585 4904 scope.go:122] "RemoveContainer" containerID="d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09" Oct 30 19:48:02.633842 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:48:02.633825 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09\": container with ID starting with d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09 not found: ID does not exist" containerID="d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09" Oct 30 19:48:02.633904 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.633848 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09"} err="failed to get container status \"d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09\": rpc error: code = NotFound desc = could not find container \"d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09\": container with ID starting with d21c06ab243306f7c9dd62d9a66a45ccb138ff91bfb084e483d106fa87376d09 not found: ID does not exist" Oct 30 19:48:02.678781 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.678758 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk"] Oct 30 19:48:02.692346 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.692315 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-6wpzk"] Oct 30 19:48:02.741859 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.741831 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8"] Oct 30 19:48:02.747937 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:02.747919 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-cjgg8"] Oct 30 19:48:03.229162 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:03.229129 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" path="/var/lib/kubelet/pods/7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82/volumes" Oct 30 19:48:03.229637 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:03.229570 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" path="/var/lib/kubelet/pods/ac0f4d7b-c714-43f5-9bb0-58a6efb416b7/volumes" Oct 30 19:48:05.400788 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.400756 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs"] Oct 30 19:48:05.401324 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.401003 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" podUID="d61769fc-6819-43b7-ab18-4b49679630b8" containerName="maas-api" containerID="cri-o://8c9a2f30d6f3b4754978331e6e79a5b3abec0505921d7fd30d3ff5f60b976a7e" gracePeriod=30 Oct 30 19:48:05.665227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.664857 4904 generic.go:356] "Generic (PLEG): container finished" podID="d61769fc-6819-43b7-ab18-4b49679630b8" containerID="8c9a2f30d6f3b4754978331e6e79a5b3abec0505921d7fd30d3ff5f60b976a7e" exitCode=0 Oct 30 19:48:05.665227 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.665084 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" event={"ID":"d61769fc-6819-43b7-ab18-4b49679630b8","Type":"ContainerDied","Data":"8c9a2f30d6f3b4754978331e6e79a5b3abec0505921d7fd30d3ff5f60b976a7e"} Oct 30 19:48:05.715466 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.715440 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:48:05.808520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.794504 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61769fc-6819-43b7-ab18-4b49679630b8-kube-api-access-dnskh\" (UniqueName: \"kubernetes.io/projected/d61769fc-6819-43b7-ab18-4b49679630b8-kube-api-access-dnskh\") pod \"d61769fc-6819-43b7-ab18-4b49679630b8\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " Oct 30 19:48:05.808520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.794588 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-metrics-certs\") pod \"d61769fc-6819-43b7-ab18-4b49679630b8\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " Oct 30 19:48:05.808520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.794633 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-maas-api-tls\") pod \"d61769fc-6819-43b7-ab18-4b49679630b8\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " Oct 30 19:48:05.808520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.794707 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61769fc-6819-43b7-ab18-4b49679630b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d61769fc-6819-43b7-ab18-4b49679630b8-openshift-service-ca\") pod \"d61769fc-6819-43b7-ab18-4b49679630b8\" (UID: \"d61769fc-6819-43b7-ab18-4b49679630b8\") " Oct 30 19:48:05.808520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.795497 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61769fc-6819-43b7-ab18-4b49679630b8-openshift-service-ca" pod "d61769fc-6819-43b7-ab18-4b49679630b8" (UID: "d61769fc-6819-43b7-ab18-4b49679630b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:05.808520 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.799037 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-maas-api-tls" pod "d61769fc-6819-43b7-ab18-4b49679630b8" (UID: "d61769fc-6819-43b7-ab18-4b49679630b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:05.812344 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.812313 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61769fc-6819-43b7-ab18-4b49679630b8-kube-api-access-dnskh" pod "d61769fc-6819-43b7-ab18-4b49679630b8" (UID: "d61769fc-6819-43b7-ab18-4b49679630b8"). InnerVolumeSpecName "kube-api-access-dnskh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:05.813000 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.812973 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-metrics-certs" pod "d61769fc-6819-43b7-ab18-4b49679630b8" (UID: "d61769fc-6819-43b7-ab18-4b49679630b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:05.896230 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.896200 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnskh\" (UniqueName: \"kubernetes.io/projected/d61769fc-6819-43b7-ab18-4b49679630b8-kube-api-access-dnskh\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:05.896230 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.896225 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:05.896230 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.896234 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d61769fc-6819-43b7-ab18-4b49679630b8-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:05.896468 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:05.896243 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d61769fc-6819-43b7-ab18-4b49679630b8-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:06.673028 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:06.672999 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" Oct 30 19:48:06.673517 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:06.673041 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs" event={"ID":"d61769fc-6819-43b7-ab18-4b49679630b8","Type":"ContainerDied","Data":"7263037b6485cc0c2fb20ed033a8b16c5af5e40fb06f9c376bab2a0dec0cad74"} Oct 30 19:48:06.673517 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:06.673097 4904 scope.go:122] "RemoveContainer" containerID="8c9a2f30d6f3b4754978331e6e79a5b3abec0505921d7fd30d3ff5f60b976a7e" Oct 30 19:48:06.728457 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:06.728432 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs"] Oct 30 19:48:06.742022 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:06.741990 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d797d2-749cb9b467-kp6bs"] Oct 30 19:48:07.221972 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:07.221940 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61769fc-6819-43b7-ab18-4b49679630b8" path="/var/lib/kubelet/pods/d61769fc-6819-43b7-ab18-4b49679630b8/volumes" Oct 30 19:48:15.930873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:15.930840 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h"] Oct 30 19:48:15.931500 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:15.931132 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" containerID="cri-o://ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd" gracePeriod=30 Oct 30 19:48:21.254891 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.254864 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:48:21.354986 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.354959 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-istio-token\" (UniqueName: \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-istio-token\") pod \"4e564002-a4b1-4f14-9986-85fd4290caed\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " Oct 30 19:48:21.355222 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.355032 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-kube-api-access-2427l\" (UniqueName: \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-kube-api-access-2427l\") pod \"4e564002-a4b1-4f14-9986-85fd4290caed\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " Oct 30 19:48:21.355222 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.355063 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-envoy\") pod \"4e564002-a4b1-4f14-9986-85fd4290caed\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " Oct 30 19:48:21.355222 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.355121 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e564002-a4b1-4f14-9986-85fd4290caed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e564002-a4b1-4f14-9986-85fd4290caed-istiod-ca-cert\") pod \"4e564002-a4b1-4f14-9986-85fd4290caed\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " Oct 30 19:48:21.355222 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.355193 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-credential-socket\") pod \"4e564002-a4b1-4f14-9986-85fd4290caed\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " Oct 30 19:48:21.355222 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.355219 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e564002-a4b1-4f14-9986-85fd4290caed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e564002-a4b1-4f14-9986-85fd4290caed-istio-podinfo\") pod \"4e564002-a4b1-4f14-9986-85fd4290caed\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " Oct 30 19:48:21.355531 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.355263 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-socket\") pod \"4e564002-a4b1-4f14-9986-85fd4290caed\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " Oct 30 19:48:21.355531 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.355340 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-certs\") pod \"4e564002-a4b1-4f14-9986-85fd4290caed\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " Oct 30 19:48:21.355531 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.355368 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-data\") pod \"4e564002-a4b1-4f14-9986-85fd4290caed\" (UID: \"4e564002-a4b1-4f14-9986-85fd4290caed\") " Oct 30 19:48:21.356342 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.356298 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-certs" pod "4e564002-a4b1-4f14-9986-85fd4290caed" (UID: "4e564002-a4b1-4f14-9986-85fd4290caed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:21.356499 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.356343 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-credential-socket" pod "4e564002-a4b1-4f14-9986-85fd4290caed" (UID: "4e564002-a4b1-4f14-9986-85fd4290caed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:21.356585 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.356561 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e564002-a4b1-4f14-9986-85fd4290caed-istiod-ca-cert" pod "4e564002-a4b1-4f14-9986-85fd4290caed" (UID: "4e564002-a4b1-4f14-9986-85fd4290caed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:21.356710 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.356624 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-data" pod "4e564002-a4b1-4f14-9986-85fd4290caed" (UID: "4e564002-a4b1-4f14-9986-85fd4290caed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:21.356710 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.356688 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-socket" pod "4e564002-a4b1-4f14-9986-85fd4290caed" (UID: "4e564002-a4b1-4f14-9986-85fd4290caed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:21.358948 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.358927 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-istio-token" pod "4e564002-a4b1-4f14-9986-85fd4290caed" (UID: "4e564002-a4b1-4f14-9986-85fd4290caed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:21.359154 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.359133 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-envoy" pod "4e564002-a4b1-4f14-9986-85fd4290caed" (UID: "4e564002-a4b1-4f14-9986-85fd4290caed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:21.359333 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.359300 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-kube-api-access-2427l" pod "4e564002-a4b1-4f14-9986-85fd4290caed" (UID: "4e564002-a4b1-4f14-9986-85fd4290caed"). InnerVolumeSpecName "kube-api-access-2427l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:21.359716 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.359696 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e564002-a4b1-4f14-9986-85fd4290caed-istio-podinfo" pod "4e564002-a4b1-4f14-9986-85fd4290caed" (UID: "4e564002-a4b1-4f14-9986-85fd4290caed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:48:21.456670 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.456641 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:21.456670 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.456670 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e564002-a4b1-4f14-9986-85fd4290caed-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:21.456874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.456683 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:21.456874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.456694 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:21.456874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.456705 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:21.456874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.456716 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:21.456874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.456728 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2427l\" (UniqueName: \"kubernetes.io/projected/4e564002-a4b1-4f14-9986-85fd4290caed-kube-api-access-2427l\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:21.456874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.456740 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e564002-a4b1-4f14-9986-85fd4290caed-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:21.456874 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.456751 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e564002-a4b1-4f14-9986-85fd4290caed-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:21.778904 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.778862 4904 generic.go:356] "Generic (PLEG): container finished" podID="4e564002-a4b1-4f14-9986-85fd4290caed" containerID="ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd" exitCode=0 Oct 30 19:48:21.779094 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.778916 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" Oct 30 19:48:21.779094 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.778951 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" event={"ID":"4e564002-a4b1-4f14-9986-85fd4290caed","Type":"ContainerDied","Data":"ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd"} Oct 30 19:48:21.779094 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.779010 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h" event={"ID":"4e564002-a4b1-4f14-9986-85fd4290caed","Type":"ContainerDied","Data":"d63584e02ac3fbab25124304414110adba06217ae973c50ca08f796830cc9a95"} Oct 30 19:48:21.779094 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.779033 4904 scope.go:122] "RemoveContainer" containerID="ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd" Oct 30 19:48:21.789548 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.789530 4904 scope.go:122] "RemoveContainer" containerID="ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd" Oct 30 19:48:21.789802 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:48:21.789785 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd\": container with ID starting with ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd not found: ID does not exist" containerID="ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd" Oct 30 19:48:21.789873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.789810 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd"} err="failed to get container status \"ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd\": rpc error: code = NotFound desc = could not find container \"ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd\": container with ID starting with ef188c29a73a330d0bc370e03ddf8325e87d91747a808c8a925b5a38292c36fd not found: ID does not exist" Oct 30 19:48:21.850389 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.850360 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h"] Oct 30 19:48:21.862275 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:21.862253 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-hsf7h"] Oct 30 19:48:23.221213 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:23.221180 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" path="/var/lib/kubelet/pods/4e564002-a4b1-4f14-9986-85fd4290caed/volumes" Oct 30 19:48:24.508800 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:48:24.508769 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:48:45.074809 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.074776 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v"] Oct 30 19:48:45.075388 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.075004 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" podUID="174adb82-e0b5-488c-bce2-91d299a6afa9" containerName="maas-api" containerID="cri-o://5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d" gracePeriod=30 Oct 30 19:48:45.379628 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.379604 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:48:45.489422 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.489391 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174adb82-e0b5-488c-bce2-91d299a6afa9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174adb82-e0b5-488c-bce2-91d299a6afa9-openshift-service-ca\") pod \"174adb82-e0b5-488c-bce2-91d299a6afa9\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " Oct 30 19:48:45.489688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.489537 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-maas-api-tls\") pod \"174adb82-e0b5-488c-bce2-91d299a6afa9\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " Oct 30 19:48:45.489688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.489578 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-metrics-certs\") pod \"174adb82-e0b5-488c-bce2-91d299a6afa9\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " Oct 30 19:48:45.489688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.489632 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174adb82-e0b5-488c-bce2-91d299a6afa9-kube-api-access-k7vzc\" (UniqueName: \"kubernetes.io/projected/174adb82-e0b5-488c-bce2-91d299a6afa9-kube-api-access-k7vzc\") pod \"174adb82-e0b5-488c-bce2-91d299a6afa9\" (UID: \"174adb82-e0b5-488c-bce2-91d299a6afa9\") " Oct 30 19:48:45.489872 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.489770 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174adb82-e0b5-488c-bce2-91d299a6afa9-openshift-service-ca" pod "174adb82-e0b5-488c-bce2-91d299a6afa9" (UID: "174adb82-e0b5-488c-bce2-91d299a6afa9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:45.489966 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.489946 4904 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174adb82-e0b5-488c-bce2-91d299a6afa9-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:45.491858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.491831 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174adb82-e0b5-488c-bce2-91d299a6afa9-kube-api-access-k7vzc" pod "174adb82-e0b5-488c-bce2-91d299a6afa9" (UID: "174adb82-e0b5-488c-bce2-91d299a6afa9"). InnerVolumeSpecName "kube-api-access-k7vzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:45.491858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.491830 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-metrics-certs" pod "174adb82-e0b5-488c-bce2-91d299a6afa9" (UID: "174adb82-e0b5-488c-bce2-91d299a6afa9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:45.491992 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.491866 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-maas-api-tls" pod "174adb82-e0b5-488c-bce2-91d299a6afa9" (UID: "174adb82-e0b5-488c-bce2-91d299a6afa9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:45.591066 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.591035 4904 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-metrics-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:45.591066 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.591062 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7vzc\" (UniqueName: \"kubernetes.io/projected/174adb82-e0b5-488c-bce2-91d299a6afa9-kube-api-access-k7vzc\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:45.591262 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.591076 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174adb82-e0b5-488c-bce2-91d299a6afa9-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:48:45.959658 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.959623 4904 generic.go:356] "Generic (PLEG): container finished" podID="174adb82-e0b5-488c-bce2-91d299a6afa9" containerID="5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d" exitCode=0 Oct 30 19:48:45.959849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.959672 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" Oct 30 19:48:45.959849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.959706 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" event={"ID":"174adb82-e0b5-488c-bce2-91d299a6afa9","Type":"ContainerDied","Data":"5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d"} Oct 30 19:48:45.959849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.959760 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v" event={"ID":"174adb82-e0b5-488c-bce2-91d299a6afa9","Type":"ContainerDied","Data":"cc4a4a1c07f083dd4d9f00cf6d36c45992e8908f4f799def3ed657c2fc49cec1"} Oct 30 19:48:45.959849 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.959785 4904 scope.go:122] "RemoveContainer" containerID="5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d" Oct 30 19:48:45.970435 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.970418 4904 scope.go:122] "RemoveContainer" containerID="5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d" Oct 30 19:48:45.970704 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:48:45.970686 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d\": container with ID starting with 5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d not found: ID does not exist" containerID="5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d" Oct 30 19:48:45.970778 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:45.970711 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d"} err="failed to get container status \"5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d\": rpc error: code = NotFound desc = could not find container \"5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d\": container with ID starting with 5f712bda457df5a7cccaa8c65c21b969cffdc022623101078e256d0ca25cff6d not found: ID does not exist" Oct 30 19:48:46.027486 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:46.024721 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v"] Oct 30 19:48:46.034538 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:46.034518 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e9de60-5bc4c66679-xvn5v"] Oct 30 19:48:47.221817 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:47.221788 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174adb82-e0b5-488c-bce2-91d299a6afa9" path="/var/lib/kubelet/pods/174adb82-e0b5-488c-bce2-91d299a6afa9/volumes" Oct 30 19:48:54.509761 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:48:54.509714 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:48:58.830679 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:58.830640 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt"] Oct 30 19:48:58.851867 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:48:58.851843 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5"] Oct 30 19:49:05.928928 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.928894 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz"] Oct 30 19:49:05.929350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929337 4904 cpu_manager.go:414] "containerMap: removing container" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" Oct 30 19:49:05.929397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929351 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" Oct 30 19:49:05.929397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929363 4904 cpu_manager.go:414] "containerMap: removing container" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerName="cleanup" Oct 30 19:49:05.929397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929368 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerName="cleanup" Oct 30 19:49:05.929397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929375 4904 cpu_manager.go:414] "containerMap: removing container" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" Oct 30 19:49:05.929397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929380 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" Oct 30 19:49:05.929397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929386 4904 cpu_manager.go:414] "containerMap: removing container" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" Oct 30 19:49:05.929397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929391 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" Oct 30 19:49:05.929397 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929400 4904 cpu_manager.go:414] "containerMap: removing container" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929405 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929411 4904 cpu_manager.go:414] "containerMap: removing container" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerName="cleanup" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929415 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerName="cleanup" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929421 4904 cpu_manager.go:414] "containerMap: removing container" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerName="cleanup" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929425 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerName="cleanup" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929434 4904 cpu_manager.go:414] "containerMap: removing container" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929438 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929446 4904 cpu_manager.go:414] "containerMap: removing container" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929450 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929457 4904 cpu_manager.go:414] "containerMap: removing container" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929462 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929469 4904 cpu_manager.go:414] "containerMap: removing container" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929473 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929482 4904 cpu_manager.go:414] "containerMap: removing container" podUID="fd683264-cd99-440c-b137-4c8b454b492f" containerName="maas-api" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929486 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd683264-cd99-440c-b137-4c8b454b492f" containerName="maas-api" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929491 4904 cpu_manager.go:414] "containerMap: removing container" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929495 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929502 4904 cpu_manager.go:414] "containerMap: removing container" podUID="174adb82-e0b5-488c-bce2-91d299a6afa9" containerName="maas-api" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929506 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174adb82-e0b5-488c-bce2-91d299a6afa9" containerName="maas-api" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929513 4904 cpu_manager.go:414] "containerMap: removing container" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929517 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929523 4904 cpu_manager.go:414] "containerMap: removing container" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerName="cleanup" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929527 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerName="cleanup" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929534 4904 cpu_manager.go:414] "containerMap: removing container" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerName="cleanup" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929538 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerName="cleanup" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929545 4904 cpu_manager.go:414] "containerMap: removing container" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929550 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929557 4904 cpu_manager.go:414] "containerMap: removing container" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929561 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929567 4904 cpu_manager.go:414] "containerMap: removing container" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929571 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929577 4904 cpu_manager.go:414] "containerMap: removing container" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929581 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929588 4904 cpu_manager.go:414] "containerMap: removing container" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerName="cleanup" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929592 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerName="cleanup" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929597 4904 cpu_manager.go:414] "containerMap: removing container" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929602 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929608 4904 cpu_manager.go:414] "containerMap: removing container" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929612 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929622 4904 cpu_manager.go:414] "containerMap: removing container" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929626 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929634 4904 cpu_manager.go:414] "containerMap: removing container" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929639 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929645 4904 cpu_manager.go:414] "containerMap: removing container" podUID="5eec7931-fe25-49a1-a11a-95040bbf7fba" containerName="maas-api" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929649 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eec7931-fe25-49a1-a11a-95040bbf7fba" containerName="maas-api" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929656 4904 cpu_manager.go:414] "containerMap: removing container" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929660 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929667 4904 cpu_manager.go:414] "containerMap: removing container" podUID="61a7275c-1864-4f89-98c7-cef6efebe04d" containerName="maas-api" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929672 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a7275c-1864-4f89-98c7-cef6efebe04d" containerName="maas-api" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929679 4904 cpu_manager.go:414] "containerMap: removing container" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929683 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" Oct 30 19:49:05.929664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929689 4904 cpu_manager.go:414] "containerMap: removing container" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929694 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929699 4904 cpu_manager.go:414] "containerMap: removing container" podUID="8d727b1d-8115-4618-9d22-2c320c1c9800" containerName="maas-api" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929703 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d727b1d-8115-4618-9d22-2c320c1c9800" containerName="maas-api" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929712 4904 cpu_manager.go:414] "containerMap: removing container" podUID="d61769fc-6819-43b7-ab18-4b49679630b8" containerName="maas-api" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929716 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61769fc-6819-43b7-ab18-4b49679630b8" containerName="maas-api" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929721 4904 cpu_manager.go:414] "containerMap: removing container" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929725 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929732 4904 cpu_manager.go:414] "containerMap: removing container" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929736 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929742 4904 cpu_manager.go:414] "containerMap: removing container" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929746 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929753 4904 cpu_manager.go:414] "containerMap: removing container" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929757 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929826 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd683264-cd99-440c-b137-4c8b454b492f" containerName="maas-api" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929836 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aad9d49-47d6-46b0-9d35-258081874d7c" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929842 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="174adb82-e0b5-488c-bce2-91d299a6afa9" containerName="maas-api" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929849 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="08afa540-378d-4cc8-8a7c-7173ddffb742" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929855 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929860 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929865 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e564002-a4b1-4f14-9986-85fd4290caed" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929870 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c54d2eb-3fa4-46ae-a0d8-b8aef7c81e82" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929875 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37ac3f1-7271-41db-8d39-9194a2820225" containerName="main" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929879 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e31c74-47bc-453f-81ef-795c60afeb3c" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929883 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929888 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="059991be-f076-488f-9aa1-70521ef6460a" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929892 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aaa2c8b-0e42-4d09-9b9c-d18fce9f7d65" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929897 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929901 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cba5790-7577-462a-aec0-759ff1d6bb40" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929907 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa2d4f9-8b9e-4915-8753-62f185d5a7ef" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929911 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eec7931-fe25-49a1-a11a-95040bbf7fba" containerName="maas-api" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929915 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929920 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c3a4f0-8862-43a6-89ac-3e6c847a6ab2" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929925 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a7275c-1864-4f89-98c7-cef6efebe04d" containerName="maas-api" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929931 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51692e8-fa0c-4042-8255-be65097e79a9" containerName="main" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929937 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0f4d7b-c714-43f5-9bb0-58a6efb416b7" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929941 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d727b1d-8115-4618-9d22-2c320c1c9800" containerName="maas-api" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929948 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61769fc-6819-43b7-ab18-4b49679630b8" containerName="maas-api" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929954 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="87aab154-3c67-4749-a67d-365be3e3437d" containerName="main" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929959 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929964 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929969 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929974 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ea566e-8d44-469a-bd0b-2b2932b16679" containerName="main" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929979 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26d1d6e-fb62-4e3c-9913-2944866d3508" containerName="main" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929984 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3b2f6a-9984-4f4f-9874-24a3bf955a27" containerName="main" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929989 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e21f26d-3a3b-4913-8a9b-0f6801f53456" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.929993 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerName="cleanup" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.930000 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="c103d769-22e3-4f04-92f6-fc9677f3098b" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.930009 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e70905-a057-43ca-864c-52a11083e726" containerName="main" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.930016 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="46da1cc6-92fa-47a0-91a8-eaf26cd86427" containerName="istio-proxy" Oct 30 19:49:05.931510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.930023 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="3793771e-d940-4326-b09d-288113928d46" containerName="istio-proxy" Oct 30 19:49:05.933951 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.933268 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:05.950741 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.950717 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz"] Oct 30 19:49:05.961671 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.961654 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:49:05.961774 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.961695 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:49:05.977559 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.977528 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c59d99-54fhz\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:05.977662 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.977585 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8znp\" (UniqueName: \"kubernetes.io/projected/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-kube-api-access-k8znp\") pod \"maas-api-revoke-sub-89c59d99-54fhz\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:05.977722 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:05.977664 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c59d99-54fhz\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:06.035732 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:06.035710 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:49:06.036044 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:06.036027 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4qb5\"" Oct 30 19:49:06.078601 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:06.078578 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c59d99-54fhz\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:06.078698 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:06.078616 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8znp\" (UniqueName: \"kubernetes.io/projected/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-kube-api-access-k8znp\") pod \"maas-api-revoke-sub-89c59d99-54fhz\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:06.078698 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:06.078633 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c59d99-54fhz\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:06.079240 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:06.079219 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c59d99-54fhz\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:06.081211 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:06.081177 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c59d99-54fhz\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:06.099674 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:06.099648 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8znp\" (UniqueName: \"kubernetes.io/projected/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-kube-api-access-k8znp\") pod \"maas-api-revoke-sub-89c59d99-54fhz\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:06.246750 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:06.246693 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:06.560493 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:49:06.560459 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc04ec32_9b5c_46df_bb62_acb3ef50c4b6.slice/crio-ee1daa96f1ed4f9846c74531cd07492013e9210112fe3c6e7be6912258ff846b WatchSource:0}: Error finding container ee1daa96f1ed4f9846c74531cd07492013e9210112fe3c6e7be6912258ff846b: Status 404 returned error can't find the container with id ee1daa96f1ed4f9846c74531cd07492013e9210112fe3c6e7be6912258ff846b Oct 30 19:49:06.560648 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:06.560626 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz"] Oct 30 19:49:07.094872 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:07.094797 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" event={"ID":"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6","Type":"ContainerStarted","Data":"b596790be7def53e5a1bbcca02659c8435917e2b0aa3b85742e95d319af6b1f0"} Oct 30 19:49:07.094872 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:07.094849 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" event={"ID":"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6","Type":"ContainerStarted","Data":"ee1daa96f1ed4f9846c74531cd07492013e9210112fe3c6e7be6912258ff846b"} Oct 30 19:49:07.146213 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:07.146168 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" podStartSLOduration=2.146155415 podStartE2EDuration="2.146155415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:07.142344182 +0000 UTC m=+1708.055948800" watchObservedRunningTime="2026-10-30 19:49:07.146155415 +0000 UTC m=+1708.059760078" Oct 30 19:49:08.103242 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:08.103203 4904 generic.go:356] "Generic (PLEG): container finished" podID="dc04ec32-9b5c-46df-bb62-acb3ef50c4b6" containerID="b596790be7def53e5a1bbcca02659c8435917e2b0aa3b85742e95d319af6b1f0" exitCode=0 Oct 30 19:49:08.103634 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:08.103277 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" event={"ID":"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6","Type":"ContainerDied","Data":"b596790be7def53e5a1bbcca02659c8435917e2b0aa3b85742e95d319af6b1f0"} Oct 30 19:49:09.297539 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:09.297507 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:09.410299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:09.410216 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-xks-ca\") pod \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " Oct 30 19:49:09.410299 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:09.410268 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-kube-api-access-k8znp\" (UniqueName: \"kubernetes.io/projected/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-kube-api-access-k8znp\") pod \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " Oct 30 19:49:09.410550 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:09.410324 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-openshift-ca\") pod \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\" (UID: \"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6\") " Oct 30 19:49:09.410974 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:09.410929 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-openshift-ca" pod "dc04ec32-9b5c-46df-bb62-acb3ef50c4b6" (UID: "dc04ec32-9b5c-46df-bb62-acb3ef50c4b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:09.412586 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:09.412557 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-xks-ca" pod "dc04ec32-9b5c-46df-bb62-acb3ef50c4b6" (UID: "dc04ec32-9b5c-46df-bb62-acb3ef50c4b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:09.412715 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:09.412618 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-kube-api-access-k8znp" pod "dc04ec32-9b5c-46df-bb62-acb3ef50c4b6" (UID: "dc04ec32-9b5c-46df-bb62-acb3ef50c4b6"). InnerVolumeSpecName "kube-api-access-k8znp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:09.512085 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:09.512052 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-xks-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.512085 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:09.512081 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8znp\" (UniqueName: \"kubernetes.io/projected/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-kube-api-access-k8znp\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.512085 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:09.512091 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6-maas-api-openshift-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:10.118798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:10.118761 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" event={"ID":"dc04ec32-9b5c-46df-bb62-acb3ef50c4b6","Type":"ContainerDied","Data":"ee1daa96f1ed4f9846c74531cd07492013e9210112fe3c6e7be6912258ff846b"} Oct 30 19:49:10.118798 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:10.118794 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz" Oct 30 19:49:10.119013 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:10.118814 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee1daa96f1ed4f9846c74531cd07492013e9210112fe3c6e7be6912258ff846b" Oct 30 19:49:13.851873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:13.851838 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" containerID="cri-o://e6754d7e0f1590d578058551f8408effb6c5e10f920ebf47522bcfdd6387c343" gracePeriod=45 Oct 30 19:49:13.874510 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:13.874481 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" containerID="cri-o://775f2663ece223f437cd77fa72248ae21a000fdc2545be0566ff660d51b2076e" gracePeriod=45 Oct 30 19:49:14.017255 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.016919 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": dial tcp 10.132.0.79:8000: connect: connection refused" Oct 30 19:49:14.163002 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.162965 4904 generic.go:356] "Generic (PLEG): container finished" podID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerID="e6754d7e0f1590d578058551f8408effb6c5e10f920ebf47522bcfdd6387c343" exitCode=0 Oct 30 19:49:14.163170 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.163137 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" event={"ID":"40606890-819d-40e3-ba76-cdd9c1be64fd","Type":"ContainerDied","Data":"e6754d7e0f1590d578058551f8408effb6c5e10f920ebf47522bcfdd6387c343"} Oct 30 19:49:14.164833 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.164806 4904 generic.go:356] "Generic (PLEG): container finished" podID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerID="775f2663ece223f437cd77fa72248ae21a000fdc2545be0566ff660d51b2076e" exitCode=0 Oct 30 19:49:14.164961 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.164950 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" event={"ID":"63ce9a7b-eec0-40d9-b0f7-829f261be3e5","Type":"ContainerDied","Data":"775f2663ece223f437cd77fa72248ae21a000fdc2545be0566ff660d51b2076e"} Oct 30 19:49:14.185473 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.185446 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:49:14.189702 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.189682 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262312 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-home\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-home\") pod \"40606890-819d-40e3-ba76-cdd9c1be64fd\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262359 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-dshm\") pod \"40606890-819d-40e3-ba76-cdd9c1be64fd\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262453 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-model-cache\") pod \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262498 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-kube-api-access-bjj7g\" (UniqueName: \"kubernetes.io/projected/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-kube-api-access-bjj7g\") pod \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262537 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-home\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-home\") pod \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262592 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40606890-819d-40e3-ba76-cdd9c1be64fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/40606890-819d-40e3-ba76-cdd9c1be64fd-tls-certs\") pod \"40606890-819d-40e3-ba76-cdd9c1be64fd\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262642 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-tmp-dir\") pod \"40606890-819d-40e3-ba76-cdd9c1be64fd\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262671 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-model-cache\") pod \"40606890-819d-40e3-ba76-cdd9c1be64fd\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262704 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tmp-dir\") pod \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262731 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tls-certs\") pod \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262773 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-dshm\") pod \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\" (UID: \"63ce9a7b-eec0-40d9-b0f7-829f261be3e5\") " Oct 30 19:49:14.263530 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.262833 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40606890-819d-40e3-ba76-cdd9c1be64fd-kube-api-access-2vn7k\" (UniqueName: \"kubernetes.io/projected/40606890-819d-40e3-ba76-cdd9c1be64fd-kube-api-access-2vn7k\") pod \"40606890-819d-40e3-ba76-cdd9c1be64fd\" (UID: \"40606890-819d-40e3-ba76-cdd9c1be64fd\") " Oct 30 19:49:14.264370 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.264331 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tmp-dir" pod "63ce9a7b-eec0-40d9-b0f7-829f261be3e5" (UID: "63ce9a7b-eec0-40d9-b0f7-829f261be3e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:14.265048 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.264629 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-model-cache" pod "40606890-819d-40e3-ba76-cdd9c1be64fd" (UID: "40606890-819d-40e3-ba76-cdd9c1be64fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:14.265048 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.264874 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-tmp-dir" pod "40606890-819d-40e3-ba76-cdd9c1be64fd" (UID: "40606890-819d-40e3-ba76-cdd9c1be64fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:14.265048 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.264993 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-model-cache" pod "63ce9a7b-eec0-40d9-b0f7-829f261be3e5" (UID: "63ce9a7b-eec0-40d9-b0f7-829f261be3e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:14.265404 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.265371 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-home" pod "63ce9a7b-eec0-40d9-b0f7-829f261be3e5" (UID: "63ce9a7b-eec0-40d9-b0f7-829f261be3e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:14.265758 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.265643 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-home" pod "40606890-819d-40e3-ba76-cdd9c1be64fd" (UID: "40606890-819d-40e3-ba76-cdd9c1be64fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:14.266315 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.266278 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40606890-819d-40e3-ba76-cdd9c1be64fd-kube-api-access-2vn7k" pod "40606890-819d-40e3-ba76-cdd9c1be64fd" (UID: "40606890-819d-40e3-ba76-cdd9c1be64fd"). InnerVolumeSpecName "kube-api-access-2vn7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:14.268245 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.268219 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40606890-819d-40e3-ba76-cdd9c1be64fd-tls-certs" pod "40606890-819d-40e3-ba76-cdd9c1be64fd" (UID: "40606890-819d-40e3-ba76-cdd9c1be64fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:14.268245 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.268230 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tls-certs" pod "63ce9a7b-eec0-40d9-b0f7-829f261be3e5" (UID: "63ce9a7b-eec0-40d9-b0f7-829f261be3e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:14.268414 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.268338 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-kube-api-access-bjj7g" pod "63ce9a7b-eec0-40d9-b0f7-829f261be3e5" (UID: "63ce9a7b-eec0-40d9-b0f7-829f261be3e5"). InnerVolumeSpecName "kube-api-access-bjj7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:14.268414 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.268338 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-dshm" pod "40606890-819d-40e3-ba76-cdd9c1be64fd" (UID: "40606890-819d-40e3-ba76-cdd9c1be64fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:14.270057 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.270038 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-dshm" pod "63ce9a7b-eec0-40d9-b0f7-829f261be3e5" (UID: "63ce9a7b-eec0-40d9-b0f7-829f261be3e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:14.364776 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364741 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364776 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364780 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjj7g\" (UniqueName: \"kubernetes.io/projected/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-kube-api-access-bjj7g\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364795 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364808 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40606890-819d-40e3-ba76-cdd9c1be64fd-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364822 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364834 4904 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364848 4904 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364859 4904 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364870 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63ce9a7b-eec0-40d9-b0f7-829f261be3e5-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364883 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vn7k\" (UniqueName: \"kubernetes.io/projected/40606890-819d-40e3-ba76-cdd9c1be64fd-kube-api-access-2vn7k\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364896 4904 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:14.364975 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:14.364910 4904 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40606890-819d-40e3-ba76-cdd9c1be64fd-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:15.172532 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:15.172501 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" Oct 30 19:49:15.173002 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:15.172527 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt" event={"ID":"40606890-819d-40e3-ba76-cdd9c1be64fd","Type":"ContainerDied","Data":"3375ff0ec6090c9a99e9eff6d58a4aaa78c391c586cbe1cc90c79e8d25bfc125"} Oct 30 19:49:15.173002 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:15.172579 4904 scope.go:122] "RemoveContainer" containerID="e6754d7e0f1590d578058551f8408effb6c5e10f920ebf47522bcfdd6387c343" Oct 30 19:49:15.174139 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:15.174094 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" event={"ID":"63ce9a7b-eec0-40d9-b0f7-829f261be3e5","Type":"ContainerDied","Data":"a80d5d110651e38ec233ed76564eb161e9321e0ecd24c3ceb86d8c2a9b8fb8be"} Oct 30 19:49:15.174244 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:15.174185 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5" Oct 30 19:49:15.184037 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:15.184023 4904 scope.go:122] "RemoveContainer" containerID="775f2663ece223f437cd77fa72248ae21a000fdc2545be0566ff660d51b2076e" Oct 30 19:49:15.228795 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:15.228778 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5"] Oct 30 19:49:15.250678 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:15.250655 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/simulator-w5-544fc8-kserve-65bdc55656-nqqk5"] Oct 30 19:49:15.279182 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:15.279163 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt"] Oct 30 19:49:15.283688 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:15.283671 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-w5-544fc8-kserve-5cfddc9444-hd7gt"] Oct 30 19:49:17.220642 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:17.220611 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" path="/var/lib/kubelet/pods/40606890-819d-40e3-ba76-cdd9c1be64fd/volumes" Oct 30 19:49:17.221018 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:17.220997 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" path="/var/lib/kubelet/pods/63ce9a7b-eec0-40d9-b0f7-829f261be3e5/volumes" Oct 30 19:49:24.510654 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:49:24.510603 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:49:28.605272 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:28.605238 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8"] Oct 30 19:49:28.606195 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:28.606158 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" containerID="cri-o://e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f" gracePeriod=30 Oct 30 19:49:33.951644 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:33.951303 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:49:34.053509 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.053465 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-podinfo\") pod \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " Oct 30 19:49:34.053777 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.053759 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-data\") pod \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " Oct 30 19:49:34.054303 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.054175 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-data" pod "65b61c1d-ecfb-484b-bc72-6b644dd10b8d" (UID: "65b61c1d-ecfb-484b-bc72-6b644dd10b8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:34.054665 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.054628 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-credential-socket" pod "65b61c1d-ecfb-484b-bc72-6b644dd10b8d" (UID: "65b61c1d-ecfb-484b-bc72-6b644dd10b8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:34.054775 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.054255 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-credential-socket\") pod \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " Oct 30 19:49:34.054915 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.054899 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-kube-api-access-r5d54\" (UniqueName: \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-kube-api-access-r5d54\") pod \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " Oct 30 19:49:34.055419 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.055402 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-token\" (UniqueName: \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-token\") pod \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " Oct 30 19:49:34.055873 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.055856 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-socket\") pod \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " Oct 30 19:49:34.056064 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.056048 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istiod-ca-cert\") pod \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " Oct 30 19:49:34.056690 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.056202 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-socket" pod "65b61c1d-ecfb-484b-bc72-6b644dd10b8d" (UID: "65b61c1d-ecfb-484b-bc72-6b644dd10b8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:34.056690 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.056501 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istiod-ca-cert" pod "65b61c1d-ecfb-484b-bc72-6b644dd10b8d" (UID: "65b61c1d-ecfb-484b-bc72-6b644dd10b8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:34.056901 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.056884 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-certs\") pod \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " Oct 30 19:49:34.057401 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.057268 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-certs" pod "65b61c1d-ecfb-484b-bc72-6b644dd10b8d" (UID: "65b61c1d-ecfb-484b-bc72-6b644dd10b8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:34.057506 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.057456 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-podinfo" pod "65b61c1d-ecfb-484b-bc72-6b644dd10b8d" (UID: "65b61c1d-ecfb-484b-bc72-6b644dd10b8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:49:34.057607 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.057343 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-envoy\") pod \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\" (UID: \"65b61c1d-ecfb-484b-bc72-6b644dd10b8d\") " Oct 30 19:49:34.058016 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.057998 4904 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:34.058150 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.058134 4904 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:34.058260 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.058248 4904 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:34.058350 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.058339 4904 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:34.058442 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.058431 4904 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:34.058534 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.058523 4904 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:34.066452 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.066419 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-kube-api-access-r5d54" pod "65b61c1d-ecfb-484b-bc72-6b644dd10b8d" (UID: "65b61c1d-ecfb-484b-bc72-6b644dd10b8d"). InnerVolumeSpecName "kube-api-access-r5d54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:34.066565 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.066541 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-envoy" pod "65b61c1d-ecfb-484b-bc72-6b644dd10b8d" (UID: "65b61c1d-ecfb-484b-bc72-6b644dd10b8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:34.074919 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.074896 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-token" pod "65b61c1d-ecfb-484b-bc72-6b644dd10b8d" (UID: "65b61c1d-ecfb-484b-bc72-6b644dd10b8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:34.159877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.159847 4904 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:34.159877 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.159874 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5d54\" (UniqueName: \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-kube-api-access-r5d54\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:34.160050 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.159889 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b61c1d-ecfb-484b-bc72-6b644dd10b8d-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:49:34.302024 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.301766 4904 generic.go:356] "Generic (PLEG): container finished" podID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerID="e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f" exitCode=0 Oct 30 19:49:34.302024 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.301820 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" Oct 30 19:49:34.302024 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.301903 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" event={"ID":"65b61c1d-ecfb-484b-bc72-6b644dd10b8d","Type":"ContainerDied","Data":"e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f"} Oct 30 19:49:34.302024 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.301952 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8" event={"ID":"65b61c1d-ecfb-484b-bc72-6b644dd10b8d","Type":"ContainerDied","Data":"cd5d979fcfdda3a583040875a25390ec5affb4aaede5f55759c516b478640d3d"} Oct 30 19:49:34.302024 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.301981 4904 scope.go:122] "RemoveContainer" containerID="e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f" Oct 30 19:49:34.316823 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.316801 4904 scope.go:122] "RemoveContainer" containerID="e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f" Oct 30 19:49:34.317149 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:49:34.317126 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f\": container with ID starting with e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f not found: ID does not exist" containerID="e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f" Oct 30 19:49:34.317243 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.317154 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f"} err="failed to get container status \"e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f\": rpc error: code = NotFound desc = could not find container \"e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f\": container with ID starting with e76eb2e2d99cfa7d8fc354ce93c550948b63e5dd327888405dc6317e19ca121f not found: ID does not exist" Oct 30 19:49:34.360663 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.360568 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8"] Oct 30 19:49:34.366684 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:34.366665 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-pl2n8"] Oct 30 19:49:35.264801 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:49:35.264766 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" path="/var/lib/kubelet/pods/65b61c1d-ecfb-484b-bc72-6b644dd10b8d/volumes" Oct 30 19:49:54.508053 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:49:54.507968 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:50:23.108628 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.108598 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx"] Oct 30 19:50:23.109097 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.108802 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" podUID="1162a95f-a50d-4cc6-b0f7-835385f46757" containerName="discovery" containerID="cri-o://73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad" gracePeriod=30 Oct 30 19:50:23.380133 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.380093 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:50:23.505604 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.505570 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1162a95f-a50d-4cc6-b0f7-835385f46757-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1162a95f-a50d-4cc6-b0f7-835385f46757-local-certs\") pod \"1162a95f-a50d-4cc6-b0f7-835385f46757\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " Oct 30 19:50:23.505785 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.505672 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-dns-cert\") pod \"1162a95f-a50d-4cc6-b0f7-835385f46757\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " Oct 30 19:50:23.505785 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.505696 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-cacerts\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-cacerts\") pod \"1162a95f-a50d-4cc6-b0f7-835385f46757\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " Oct 30 19:50:23.505785 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.505738 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-token\" (UniqueName: \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-token\") pod \"1162a95f-a50d-4cc6-b0f7-835385f46757\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " Oct 30 19:50:23.505785 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.505764 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-ca-configmap\") pod \"1162a95f-a50d-4cc6-b0f7-835385f46757\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " Oct 30 19:50:23.506010 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.505858 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-kubeconfig\") pod \"1162a95f-a50d-4cc6-b0f7-835385f46757\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " Oct 30 19:50:23.506010 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.505892 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-kube-api-access-gf7d4\" (UniqueName: \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-kube-api-access-gf7d4\") pod \"1162a95f-a50d-4cc6-b0f7-835385f46757\" (UID: \"1162a95f-a50d-4cc6-b0f7-835385f46757\") " Oct 30 19:50:23.506302 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.506264 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-ca-configmap" pod "1162a95f-a50d-4cc6-b0f7-835385f46757" (UID: "1162a95f-a50d-4cc6-b0f7-835385f46757"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:23.508230 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.508201 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-dns-cert" pod "1162a95f-a50d-4cc6-b0f7-835385f46757" (UID: "1162a95f-a50d-4cc6-b0f7-835385f46757"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:23.508495 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.508464 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-kube-api-access-gf7d4" pod "1162a95f-a50d-4cc6-b0f7-835385f46757" (UID: "1162a95f-a50d-4cc6-b0f7-835385f46757"). InnerVolumeSpecName "kube-api-access-gf7d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:23.508664 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.508639 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1162a95f-a50d-4cc6-b0f7-835385f46757-local-certs" pod "1162a95f-a50d-4cc6-b0f7-835385f46757" (UID: "1162a95f-a50d-4cc6-b0f7-835385f46757"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:23.508751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.508638 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-kubeconfig" pod "1162a95f-a50d-4cc6-b0f7-835385f46757" (UID: "1162a95f-a50d-4cc6-b0f7-835385f46757"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:23.508751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.508692 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-cacerts" pod "1162a95f-a50d-4cc6-b0f7-835385f46757" (UID: "1162a95f-a50d-4cc6-b0f7-835385f46757"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:23.508751 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.508714 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-token" pod "1162a95f-a50d-4cc6-b0f7-835385f46757" (UID: "1162a95f-a50d-4cc6-b0f7-835385f46757"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:23.607124 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.607078 4904 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-dns-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:50:23.607249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.607130 4904 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-cacerts\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:50:23.607249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.607146 4904 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:50:23.607249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.607160 4904 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-csr-ca-configmap\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:50:23.607249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.607174 4904 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1162a95f-a50d-4cc6-b0f7-835385f46757-istio-kubeconfig\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:50:23.607249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.607184 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf7d4\" (UniqueName: \"kubernetes.io/projected/1162a95f-a50d-4cc6-b0f7-835385f46757-kube-api-access-gf7d4\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:50:23.607249 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.607193 4904 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1162a95f-a50d-4cc6-b0f7-835385f46757-local-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:50:23.642529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.642475 4904 generic.go:356] "Generic (PLEG): container finished" podID="1162a95f-a50d-4cc6-b0f7-835385f46757" containerID="73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad" exitCode=0 Oct 30 19:50:23.642529 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.642520 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" event={"ID":"1162a95f-a50d-4cc6-b0f7-835385f46757","Type":"ContainerDied","Data":"73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad"} Oct 30 19:50:23.642695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.642529 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" Oct 30 19:50:23.642695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.642557 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx" event={"ID":"1162a95f-a50d-4cc6-b0f7-835385f46757","Type":"ContainerDied","Data":"832080d7f99a978bbb9f5b4651251145b86e2ade037c8e55f9123e7439a898a0"} Oct 30 19:50:23.642695 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.642576 4904 scope.go:122] "RemoveContainer" containerID="73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad" Oct 30 19:50:23.657293 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.657270 4904 scope.go:122] "RemoveContainer" containerID="73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad" Oct 30 19:50:23.657601 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:50:23.657582 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad\": container with ID starting with 73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad not found: ID does not exist" containerID="73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad" Oct 30 19:50:23.657680 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.657610 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad"} err="failed to get container status \"73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad\": rpc error: code = NotFound desc = could not find container \"73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad\": container with ID starting with 73f6f2937f5ac8580d4d3a590eba61190cbaeb5e7245d915ab4de66a474488ad not found: ID does not exist" Oct 30 19:50:23.703231 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.703206 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx"] Oct 30 19:50:23.716382 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:23.716352 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zf9jx"] Oct 30 19:50:24.508450 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:50:24.508415 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:50:25.220605 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:25.220571 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1162a95f-a50d-4cc6-b0f7-835385f46757" path="/var/lib/kubelet/pods/1162a95f-a50d-4cc6-b0f7-835385f46757/volumes" Oct 30 19:50:39.598219 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:39.598193 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:50:39.607483 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:39.607457 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:50:39.615603 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:39.615585 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:50:39.624705 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:50:39.624688 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:50:54.607922 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:50:54.607881 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:51:24.508936 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:51:24.508855 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:51:54.522851 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:51:54.522809 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:52:24.508679 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:52:24.508641 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:52:48.818130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:52:48.814242 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ggdsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:52:48.818130 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:52:48.814318 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ggdsj" podUID="0c8a2a4b-6240-40fa-a873-d2300e317a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:54.509386 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:52:54.509352 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:53:24.509743 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:53:24.509706 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:53:54.596369 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:53:54.596327 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:54:10.035663 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:54:10.035624 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz"] Oct 30 19:54:10.048938 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:54:10.048910 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c59d99-54fhz"] Oct 30 19:54:11.220088 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:54:11.220057 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc04ec32-9b5c-46df-bb62-acb3ef50c4b6" path="/var/lib/kubelet/pods/dc04ec32-9b5c-46df-bb62-acb3ef50c4b6/volumes" Oct 30 19:54:24.516052 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:54:24.515947 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:54:54.509742 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:54:54.509707 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:55:24.510345 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:55:24.510296 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:55:39.647234 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:55:39.647201 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:55:39.659224 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:55:39.659201 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 19:55:39.669191 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:55:39.669170 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:55:39.677858 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:55:39.677839 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 19:55:41.643117 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:55:41.643066 4904 scope.go:122] "RemoveContainer" containerID="b596790be7def53e5a1bbcca02659c8435917e2b0aa3b85742e95d319af6b1f0" Oct 30 19:55:54.510902 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:55:54.510818 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:56:24.509457 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:56:24.509420 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:56:53.350822 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:53.350775 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn"] Oct 30 19:56:53.351341 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:53.350993 4904 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" podUID="54ad1f5f-8f45-463b-9c5e-b0f9f06315c4" containerName="manager" containerID="cri-o://9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f" gracePeriod=10 Oct 30 19:56:53.615507 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:53.615449 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:56:53.699501 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:53.699472 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-kube-api-access-t4vpt\" (UniqueName: \"kubernetes.io/projected/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-kube-api-access-t4vpt\") pod \"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4\" (UID: \"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4\") " Oct 30 19:56:53.699669 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:53.699560 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-extensions-socket-volume\") pod \"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4\" (UID: \"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4\") " Oct 30 19:56:53.700032 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:53.700001 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-extensions-socket-volume" pod "54ad1f5f-8f45-463b-9c5e-b0f9f06315c4" (UID: "54ad1f5f-8f45-463b-9c5e-b0f9f06315c4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:53.701752 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:53.701724 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-kube-api-access-t4vpt" pod "54ad1f5f-8f45-463b-9c5e-b0f9f06315c4" (UID: "54ad1f5f-8f45-463b-9c5e-b0f9f06315c4"). InnerVolumeSpecName "kube-api-access-t4vpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:53.800891 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:53.800862 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4vpt\" (UniqueName: \"kubernetes.io/projected/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-kube-api-access-t4vpt\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:56:53.800891 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:53.800888 4904 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4-extensions-socket-volume\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.211659 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:54.211630 4904 generic.go:356] "Generic (PLEG): container finished" podID="54ad1f5f-8f45-463b-9c5e-b0f9f06315c4" containerID="9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f" exitCode=0 Oct 30 19:56:54.211823 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:54.211669 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" Oct 30 19:56:54.211823 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:54.211710 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" event={"ID":"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4","Type":"ContainerDied","Data":"9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f"} Oct 30 19:56:54.211823 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:54.211759 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn" event={"ID":"54ad1f5f-8f45-463b-9c5e-b0f9f06315c4","Type":"ContainerDied","Data":"2ed8c7d981ffc9de4dc2ae2a8d83b612fcdd2f5e5b96013b3ef6d73081138365"} Oct 30 19:56:54.211823 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:54.211782 4904 scope.go:122] "RemoveContainer" containerID="9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f" Oct 30 19:56:54.224817 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:54.224801 4904 scope.go:122] "RemoveContainer" containerID="9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f" Oct 30 19:56:54.225051 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:56:54.225033 4904 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f\": container with ID starting with 9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f not found: ID does not exist" containerID="9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f" Oct 30 19:56:54.225252 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:54.225058 4904 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f"} err="failed to get container status \"9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f\": rpc error: code = NotFound desc = could not find container \"9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f\": container with ID starting with 9b461988d98467fa51de046d4ea4d96916fd7e7a5ddb4d35bee9a75ce5024a3f not found: ID does not exist" Oct 30 19:56:54.268599 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:54.268567 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn"] Oct 30 19:56:54.272734 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:54.272711 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vtgpn"] Oct 30 19:56:54.508592 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:56:54.508507 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:56:55.220732 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:56:55.220691 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ad1f5f-8f45-463b-9c5e-b0f9f06315c4" path="/var/lib/kubelet/pods/54ad1f5f-8f45-463b-9c5e-b0f9f06315c4/volumes" Oct 30 19:57:24.508810 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:57:24.508730 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:57:54.509570 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:57:54.509533 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:57:56.527942 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.527908 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6"] Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528295 4904 cpu_manager.go:414] "containerMap: removing container" podUID="54ad1f5f-8f45-463b-9c5e-b0f9f06315c4" containerName="manager" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528306 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ad1f5f-8f45-463b-9c5e-b0f9f06315c4" containerName="manager" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528322 4904 cpu_manager.go:414] "containerMap: removing container" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528327 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528344 4904 cpu_manager.go:414] "containerMap: removing container" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerName="cleanup" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528349 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerName="cleanup" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528358 4904 cpu_manager.go:414] "containerMap: removing container" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerName="cleanup" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528362 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerName="cleanup" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528379 4904 cpu_manager.go:414] "containerMap: removing container" podUID="1162a95f-a50d-4cc6-b0f7-835385f46757" containerName="discovery" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528387 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1162a95f-a50d-4cc6-b0f7-835385f46757" containerName="discovery" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528394 4904 cpu_manager.go:414] "containerMap: removing container" podUID="dc04ec32-9b5c-46df-bb62-acb3ef50c4b6" containerName="revoke-keys" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528398 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc04ec32-9b5c-46df-bb62-acb3ef50c4b6" containerName="revoke-keys" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528407 4904 cpu_manager.go:414] "containerMap: removing container" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528412 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528421 4904 cpu_manager.go:414] "containerMap: removing container" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerName="cleanup" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528425 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d48683-e2d8-4332-acb2-abd4133fff88" containerName="cleanup" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528434 4904 cpu_manager.go:414] "containerMap: removing container" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" Oct 30 19:57:56.528459 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528438 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" Oct 30 19:57:56.529047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528492 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="40606890-819d-40e3-ba76-cdd9c1be64fd" containerName="main" Oct 30 19:57:56.529047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528504 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="05fe0b01-aaa5-4c59-9848-48c831f7dc4d" containerName="cleanup" Oct 30 19:57:56.529047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528509 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b2d89e-87fd-4ffc-a8f0-7dadfc414fb1" containerName="cleanup" Oct 30 19:57:56.529047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528515 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="1162a95f-a50d-4cc6-b0f7-835385f46757" containerName="discovery" Oct 30 19:57:56.529047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528521 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc04ec32-9b5c-46df-bb62-acb3ef50c4b6" containerName="revoke-keys" Oct 30 19:57:56.529047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528527 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ce9a7b-eec0-40d9-b0f7-829f261be3e5" containerName="main" Oct 30 19:57:56.529047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528533 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b61c1d-ecfb-484b-bc72-6b644dd10b8d" containerName="istio-proxy" Oct 30 19:57:56.529047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528538 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ad1f5f-8f45-463b-9c5e-b0f9f06315c4" containerName="manager" Oct 30 19:57:56.529047 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.528544 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca5963c-22d7-4688-b367-1aa657eb37b4" containerName="cleanup" Oct 30 19:57:56.532839 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.532823 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" Oct 30 19:57:56.544948 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.544923 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6"] Oct 30 19:57:56.555253 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.555219 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zh7c7\"" Oct 30 19:57:56.606452 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.606428 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qd9\" (UniqueName: \"kubernetes.io/projected/336f7de4-e1c6-4e67-8252-9f5299137355-kube-api-access-78qd9\") pod \"kuadrant-operator-controller-manager-678d87766f-hwhj6\" (UID: \"336f7de4-e1c6-4e67-8252-9f5299137355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" Oct 30 19:57:56.606563 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.606462 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/336f7de4-e1c6-4e67-8252-9f5299137355-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hwhj6\" (UID: \"336f7de4-e1c6-4e67-8252-9f5299137355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" Oct 30 19:57:56.707263 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.707238 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qd9\" (UniqueName: \"kubernetes.io/projected/336f7de4-e1c6-4e67-8252-9f5299137355-kube-api-access-78qd9\") pod \"kuadrant-operator-controller-manager-678d87766f-hwhj6\" (UID: \"336f7de4-e1c6-4e67-8252-9f5299137355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" Oct 30 19:57:56.707380 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.707271 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/336f7de4-e1c6-4e67-8252-9f5299137355-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hwhj6\" (UID: \"336f7de4-e1c6-4e67-8252-9f5299137355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" Oct 30 19:57:56.707595 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.707581 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/336f7de4-e1c6-4e67-8252-9f5299137355-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hwhj6\" (UID: \"336f7de4-e1c6-4e67-8252-9f5299137355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" Oct 30 19:57:56.732235 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.732207 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qd9\" (UniqueName: \"kubernetes.io/projected/336f7de4-e1c6-4e67-8252-9f5299137355-kube-api-access-78qd9\") pod \"kuadrant-operator-controller-manager-678d87766f-hwhj6\" (UID: \"336f7de4-e1c6-4e67-8252-9f5299137355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" Oct 30 19:57:56.847410 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:56.847355 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" Oct 30 19:57:57.203234 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:57.203207 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6"] Oct 30 19:57:57.204388 ip-10-0-138-21 kubenswrapper[4904]: W1030 19:57:57.204356 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336f7de4_e1c6_4e67_8252_9f5299137355.slice/crio-89cdcd9e5bdb24085ce22502899ba6ecb5afbb1d3ff178907bf7f65d083bd756 WatchSource:0}: Error finding container 89cdcd9e5bdb24085ce22502899ba6ecb5afbb1d3ff178907bf7f65d083bd756: Status 404 returned error can't find the container with id 89cdcd9e5bdb24085ce22502899ba6ecb5afbb1d3ff178907bf7f65d083bd756 Oct 30 19:57:57.570866 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:57.570780 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" event={"ID":"336f7de4-e1c6-4e67-8252-9f5299137355","Type":"ContainerStarted","Data":"fa63aebe6026de3cbc33e5b03734823f619f7a7bd093661f7d54d6404c0cc237"} Oct 30 19:57:57.570866 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:57.570834 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" event={"ID":"336f7de4-e1c6-4e67-8252-9f5299137355","Type":"ContainerStarted","Data":"89cdcd9e5bdb24085ce22502899ba6ecb5afbb1d3ff178907bf7f65d083bd756"} Oct 30 19:57:57.570866 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:57.570864 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" Oct 30 19:57:57.643531 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:57:57.643488 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" podStartSLOduration=1.6434756510000001 podStartE2EDuration="1.643475651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:57.641366964 +0000 UTC m=+2238.554971580" watchObservedRunningTime="2026-10-30 19:57:57.643475651 +0000 UTC m=+2238.557080267" Oct 30 19:58:08.578831 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:58:08.578803 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwhj6" Oct 30 19:58:24.508585 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:58:24.508548 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:58:34.840267 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:58:34.840222 4904 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8k4b6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:58:34.841011 ip-10-0-138-21 kubenswrapper[4904]: I1030 19:58:34.840979 4904 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8k4b6" podUID="8da395a9-9be0-46ba-b397-ee54354bf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:54.512640 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:58:54.512540 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:59:24.508975 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:59:24.508939 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 19:59:54.508694 ip-10-0-138-21 kubenswrapper[4904]: E1030 19:59:54.508653 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:00:00.144513 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.144475 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x"] Oct 30 20:00:00.148363 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.148341 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" Oct 30 20:00:00.172023 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.172000 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x"] Oct 30 20:00:00.181978 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.181954 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:00:00.182068 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.182010 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:00:00.206528 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.206498 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zpf6\" (UniqueName: \"kubernetes.io/projected/ab14b042-1a55-471f-bb25-e1fb63b397eb-kube-api-access-2zpf6\") pod \"maas-api-key-cleanup-29889840-4q69x\" (UID: \"ab14b042-1a55-471f-bb25-e1fb63b397eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" Oct 30 20:00:00.206623 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.206548 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab14b042-1a55-471f-bb25-e1fb63b397eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-4q69x\" (UID: \"ab14b042-1a55-471f-bb25-e1fb63b397eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" Oct 30 20:00:00.262977 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.262957 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4qb5\"" Oct 30 20:00:00.308422 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.308392 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zpf6\" (UniqueName: \"kubernetes.io/projected/ab14b042-1a55-471f-bb25-e1fb63b397eb-kube-api-access-2zpf6\") pod \"maas-api-key-cleanup-29889840-4q69x\" (UID: \"ab14b042-1a55-471f-bb25-e1fb63b397eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" Oct 30 20:00:00.308552 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.308485 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab14b042-1a55-471f-bb25-e1fb63b397eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-4q69x\" (UID: \"ab14b042-1a55-471f-bb25-e1fb63b397eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" Oct 30 20:00:00.309627 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.309597 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab14b042-1a55-471f-bb25-e1fb63b397eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-4q69x\" (UID: \"ab14b042-1a55-471f-bb25-e1fb63b397eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" Oct 30 20:00:00.336534 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.336498 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zpf6\" (UniqueName: \"kubernetes.io/projected/ab14b042-1a55-471f-bb25-e1fb63b397eb-kube-api-access-2zpf6\") pod \"maas-api-key-cleanup-29889840-4q69x\" (UID: \"ab14b042-1a55-471f-bb25-e1fb63b397eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" Oct 30 20:00:00.461477 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.461453 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" Oct 30 20:00:00.762226 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:00.762197 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x"] Oct 30 20:00:00.763280 ip-10-0-138-21 kubenswrapper[4904]: W1030 20:00:00.763251 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab14b042_1a55_471f_bb25_e1fb63b397eb.slice/crio-9bb959bd5abe6175e6fdf19edcc849012efbe55d2b5b8a234fa3889b0a2cf496 WatchSource:0}: Error finding container 9bb959bd5abe6175e6fdf19edcc849012efbe55d2b5b8a234fa3889b0a2cf496: Status 404 returned error can't find the container with id 9bb959bd5abe6175e6fdf19edcc849012efbe55d2b5b8a234fa3889b0a2cf496 Oct 30 20:00:01.329484 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:01.329405 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" event={"ID":"ab14b042-1a55-471f-bb25-e1fb63b397eb","Type":"ContainerStarted","Data":"648abde5f2e7750514b4c3648accc8397edc6930fb760cf26459e323386718db"} Oct 30 20:00:01.329484 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:01.329456 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" event={"ID":"ab14b042-1a55-471f-bb25-e1fb63b397eb","Type":"ContainerStarted","Data":"9bb959bd5abe6175e6fdf19edcc849012efbe55d2b5b8a234fa3889b0a2cf496"} Oct 30 20:00:01.372151 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:01.372084 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" podStartSLOduration=1.372071505 podStartE2EDuration="1.372071505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:01.371195064 +0000 UTC m=+2362.284799680" watchObservedRunningTime="2026-10-30 20:00:01.372071505 +0000 UTC m=+2362.285676131" Oct 30 20:00:02.336801 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:02.336752 4904 generic.go:356] "Generic (PLEG): container finished" podID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerID="648abde5f2e7750514b4c3648accc8397edc6930fb760cf26459e323386718db" exitCode=60 Oct 30 20:00:02.337298 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:02.336845 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" event={"ID":"ab14b042-1a55-471f-bb25-e1fb63b397eb","Type":"ContainerDied","Data":"648abde5f2e7750514b4c3648accc8397edc6930fb760cf26459e323386718db"} Oct 30 20:00:02.337298 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:02.337030 4904 scope.go:122] "RemoveContainer" containerID="648abde5f2e7750514b4c3648accc8397edc6930fb760cf26459e323386718db" Oct 30 20:00:03.343961 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:03.343927 4904 generic.go:356] "Generic (PLEG): container finished" podID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerID="9c49913f0a86edca9a4353acf25df710aaa6dc9a83131d908e5c123e4d38f807" exitCode=60 Oct 30 20:00:03.344446 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:03.343986 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" event={"ID":"ab14b042-1a55-471f-bb25-e1fb63b397eb","Type":"ContainerDied","Data":"9c49913f0a86edca9a4353acf25df710aaa6dc9a83131d908e5c123e4d38f807"} Oct 30 20:00:03.344446 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:03.344032 4904 scope.go:122] "RemoveContainer" containerID="648abde5f2e7750514b4c3648accc8397edc6930fb760cf26459e323386718db" Oct 30 20:00:03.344446 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:03.344251 4904 scope.go:122] "RemoveContainer" containerID="9c49913f0a86edca9a4353acf25df710aaa6dc9a83131d908e5c123e4d38f807" Oct 30 20:00:03.344618 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:00:03.344494 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-4q69x_odh-ai-gateway-infra(ab14b042-1a55-471f-bb25-e1fb63b397eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" Oct 30 20:00:04.350949 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:04.350918 4904 scope.go:122] "RemoveContainer" containerID="9c49913f0a86edca9a4353acf25df710aaa6dc9a83131d908e5c123e4d38f807" Oct 30 20:00:04.351427 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:00:04.351218 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-4q69x_odh-ai-gateway-infra(ab14b042-1a55-471f-bb25-e1fb63b397eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" Oct 30 20:00:15.215415 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:15.215385 4904 scope.go:122] "RemoveContainer" containerID="9c49913f0a86edca9a4353acf25df710aaa6dc9a83131d908e5c123e4d38f807" Oct 30 20:00:16.422830 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:16.422793 4904 generic.go:356] "Generic (PLEG): container finished" podID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerID="ea7f8a00cd42d18a8e00961830e9cae1fd65d98294d2fe57c6744a9847d8acc4" exitCode=60 Oct 30 20:00:16.423279 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:16.422861 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" event={"ID":"ab14b042-1a55-471f-bb25-e1fb63b397eb","Type":"ContainerDied","Data":"ea7f8a00cd42d18a8e00961830e9cae1fd65d98294d2fe57c6744a9847d8acc4"} Oct 30 20:00:16.423279 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:16.422924 4904 scope.go:122] "RemoveContainer" containerID="9c49913f0a86edca9a4353acf25df710aaa6dc9a83131d908e5c123e4d38f807" Oct 30 20:00:16.423279 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:16.423248 4904 scope.go:122] "RemoveContainer" containerID="ea7f8a00cd42d18a8e00961830e9cae1fd65d98294d2fe57c6744a9847d8acc4" Oct 30 20:00:16.423503 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:00:16.423477 4904 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-4q69x_odh-ai-gateway-infra(ab14b042-1a55-471f-bb25-e1fb63b397eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" Oct 30 20:00:17.474398 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:17.474371 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x"] Oct 30 20:00:17.618549 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:17.618513 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" Oct 30 20:00:17.679227 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:17.679196 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab14b042-1a55-471f-bb25-e1fb63b397eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab14b042-1a55-471f-bb25-e1fb63b397eb-maas-api-service-ca\") pod \"ab14b042-1a55-471f-bb25-e1fb63b397eb\" (UID: \"ab14b042-1a55-471f-bb25-e1fb63b397eb\") " Oct 30 20:00:17.679518 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:17.679495 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab14b042-1a55-471f-bb25-e1fb63b397eb-kube-api-access-2zpf6\" (UniqueName: \"kubernetes.io/projected/ab14b042-1a55-471f-bb25-e1fb63b397eb-kube-api-access-2zpf6\") pod \"ab14b042-1a55-471f-bb25-e1fb63b397eb\" (UID: \"ab14b042-1a55-471f-bb25-e1fb63b397eb\") " Oct 30 20:00:17.680235 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:17.680082 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab14b042-1a55-471f-bb25-e1fb63b397eb-maas-api-service-ca" pod "ab14b042-1a55-471f-bb25-e1fb63b397eb" (UID: "ab14b042-1a55-471f-bb25-e1fb63b397eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:17.680715 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:17.680689 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab14b042-1a55-471f-bb25-e1fb63b397eb-maas-api-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:00:17.682152 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:17.682127 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab14b042-1a55-471f-bb25-e1fb63b397eb-kube-api-access-2zpf6" pod "ab14b042-1a55-471f-bb25-e1fb63b397eb" (UID: "ab14b042-1a55-471f-bb25-e1fb63b397eb"). InnerVolumeSpecName "kube-api-access-2zpf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.782238 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:17.782169 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zpf6\" (UniqueName: \"kubernetes.io/projected/ab14b042-1a55-471f-bb25-e1fb63b397eb-kube-api-access-2zpf6\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:00:18.439147 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:18.439092 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" event={"ID":"ab14b042-1a55-471f-bb25-e1fb63b397eb","Type":"ContainerDied","Data":"9bb959bd5abe6175e6fdf19edcc849012efbe55d2b5b8a234fa3889b0a2cf496"} Oct 30 20:00:18.439347 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:18.439171 4904 scope.go:122] "RemoveContainer" containerID="ea7f8a00cd42d18a8e00961830e9cae1fd65d98294d2fe57c6744a9847d8acc4" Oct 30 20:00:18.439347 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:18.439123 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x" Oct 30 20:00:18.496778 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:18.496752 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x"] Oct 30 20:00:18.509546 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:18.509522 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4q69x"] Oct 30 20:00:19.222149 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:19.222051 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" path="/var/lib/kubelet/pods/ab14b042-1a55-471f-bb25-e1fb63b397eb/volumes" Oct 30 20:00:24.511005 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:00:24.510969 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:00:39.768360 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:39.765801 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 20:00:39.783905 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:39.783869 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 20:00:39.838086 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:39.838059 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 20:00:39.842379 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:00:39.842292 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 20:00:54.518282 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:00:54.518237 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:01:24.523039 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:01:24.522964 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:01:54.510264 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:01:54.510175 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:02:18.418916 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.418884 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn"] Oct 30 20:02:18.419423 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.419301 4904 cpu_manager.go:414] "containerMap: removing container" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerName="cleanup" Oct 30 20:02:18.419423 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.419312 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerName="cleanup" Oct 30 20:02:18.419423 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.419335 4904 cpu_manager.go:414] "containerMap: removing container" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerName="cleanup" Oct 30 20:02:18.419423 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.419340 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerName="cleanup" Oct 30 20:02:18.419423 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.419347 4904 cpu_manager.go:414] "containerMap: removing container" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerName="cleanup" Oct 30 20:02:18.419423 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.419352 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerName="cleanup" Oct 30 20:02:18.419423 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.419410 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerName="cleanup" Oct 30 20:02:18.419423 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.419422 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerName="cleanup" Oct 30 20:02:18.422352 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.422336 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:18.440471 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.440430 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:18.440712 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.440694 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn"] Oct 30 20:02:18.451234 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.451214 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:02:18.451234 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.451226 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:02:18.451394 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.451299 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4qb5\"" Oct 30 20:02:18.525082 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.525042 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m968q\" (UniqueName: \"kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:18.525264 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.525164 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:18.525319 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.525304 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:18.626488 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.626456 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:18.626684 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.626499 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m968q\" (UniqueName: \"kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:18.626762 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.626711 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:18.627330 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.627309 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:18.628969 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:18.628945 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:18.658725 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:02:18.658706 4904 projected.go:196] Error preparing data for projected volume kube-api-access-m968q for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 20:02:18.658847 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:02:18.658789 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q podName:356beeac-354b-475d-9e61-5fd58359ad64 nodeName:}" failed. No retries permitted until 2026-10-30 20:02:19.158769276 +0000 UTC m=+2500.072373876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m968q" (UniqueName: "kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q") pod "maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" (UID: "356beeac-354b-475d-9e61-5fd58359ad64") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 20:02:19.230175 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:19.230138 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m968q\" (UniqueName: \"kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:19.245815 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:02:19.245790 4904 projected.go:196] Error preparing data for projected volume kube-api-access-m968q for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 20:02:19.245949 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:02:19.245845 4904 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q podName:356beeac-354b-475d-9e61-5fd58359ad64 nodeName:}" failed. No retries permitted until 2026-10-30 20:02:20.245830873 +0000 UTC m=+2501.159435468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m968q" (UniqueName: "kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q") pod "maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" (UID: "356beeac-354b-475d-9e61-5fd58359ad64") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 20:02:20.340615 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:20.340575 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m968q\" (UniqueName: \"kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:20.365194 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:20.365159 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m968q\" (UniqueName: \"kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q\") pod \"maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:20.535811 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:20.535782 4904 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" secret="" err="secret \"maas-api-cleanup-dockercfg-q4qb5\" not found" Oct 30 20:02:20.535933 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:20.535840 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:20.839201 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:20.838002 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn"] Oct 30 20:02:21.404922 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:21.404838 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" event={"ID":"356beeac-354b-475d-9e61-5fd58359ad64","Type":"ContainerStarted","Data":"78ec247e6cf5ad272f5525c635a4705c93b217447f0b84beb5bb9ba78a7a9639"} Oct 30 20:02:21.404922 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:21.404892 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" event={"ID":"356beeac-354b-475d-9e61-5fd58359ad64","Type":"ContainerStarted","Data":"56e9b312891e9b409fbedac1cadc36fa91252d9fabcd58df3ad094e1b088adfe"} Oct 30 20:02:21.405373 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:21.405093 4904 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" secret="" err="secret \"maas-api-cleanup-dockercfg-q4qb5\" not found" Oct 30 20:02:21.448968 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:21.448921 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" podStartSLOduration=3.4489098719999998 podStartE2EDuration="3.448909872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:21.447725367 +0000 UTC m=+2502.361329985" watchObservedRunningTime="2026-10-30 20:02:21.448909872 +0000 UTC m=+2502.362514488" Oct 30 20:02:22.412142 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:22.412088 4904 generic.go:356] "Generic (PLEG): container finished" podID="356beeac-354b-475d-9e61-5fd58359ad64" containerID="78ec247e6cf5ad272f5525c635a4705c93b217447f0b84beb5bb9ba78a7a9639" exitCode=0 Oct 30 20:02:22.412560 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:22.412205 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" event={"ID":"356beeac-354b-475d-9e61-5fd58359ad64","Type":"ContainerDied","Data":"78ec247e6cf5ad272f5525c635a4705c93b217447f0b84beb5bb9ba78a7a9639"} Oct 30 20:02:23.556370 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:23.556346 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:23.665816 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:23.665783 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q\" (UniqueName: \"kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q\") pod \"356beeac-354b-475d-9e61-5fd58359ad64\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " Oct 30 20:02:23.666001 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:23.665828 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-xks-ca\") pod \"356beeac-354b-475d-9e61-5fd58359ad64\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " Oct 30 20:02:23.666001 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:23.665895 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-openshift-ca\") pod \"356beeac-354b-475d-9e61-5fd58359ad64\" (UID: \"356beeac-354b-475d-9e61-5fd58359ad64\") " Oct 30 20:02:23.666316 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:23.666288 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-openshift-ca" pod "356beeac-354b-475d-9e61-5fd58359ad64" (UID: "356beeac-354b-475d-9e61-5fd58359ad64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:23.668155 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:23.668135 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-xks-ca" pod "356beeac-354b-475d-9e61-5fd58359ad64" (UID: "356beeac-354b-475d-9e61-5fd58359ad64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:23.668226 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:23.668151 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q" pod "356beeac-354b-475d-9e61-5fd58359ad64" (UID: "356beeac-354b-475d-9e61-5fd58359ad64"). InnerVolumeSpecName "kube-api-access-m968q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:23.766983 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:23.766958 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-xks-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.766983 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:23.766982 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356beeac-354b-475d-9e61-5fd58359ad64-maas-api-openshift-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.767149 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:23.766991 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m968q\" (UniqueName: \"kubernetes.io/projected/356beeac-354b-475d-9e61-5fd58359ad64-kube-api-access-m968q\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:02:24.425147 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:24.425115 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" Oct 30 20:02:24.425147 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:24.425132 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn" event={"ID":"356beeac-354b-475d-9e61-5fd58359ad64","Type":"ContainerDied","Data":"56e9b312891e9b409fbedac1cadc36fa91252d9fabcd58df3ad094e1b088adfe"} Oct 30 20:02:24.425368 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:24.425185 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e9b312891e9b409fbedac1cadc36fa91252d9fabcd58df3ad094e1b088adfe" Oct 30 20:02:24.509602 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:02:24.509566 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:02:30.627799 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:30.627760 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn"] Oct 30 20:02:30.640998 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:30.640966 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8feb399-fpnsn"] Oct 30 20:02:31.223848 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:02:31.223809 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356beeac-354b-475d-9e61-5fd58359ad64" path="/var/lib/kubelet/pods/356beeac-354b-475d-9e61-5fd58359ad64/volumes" Oct 30 20:02:54.509313 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:02:54.509266 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:03:24.509148 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:03:24.509039 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:03:54.509823 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:03:54.509784 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:04:15.416277 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.416214 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr"] Oct 30 20:04:15.417270 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.417245 4904 cpu_manager.go:414] "containerMap: removing container" podUID="356beeac-354b-475d-9e61-5fd58359ad64" containerName="revoke-keys" Oct 30 20:04:15.417270 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.417272 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356beeac-354b-475d-9e61-5fd58359ad64" containerName="revoke-keys" Oct 30 20:04:15.417530 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.417516 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab14b042-1a55-471f-bb25-e1fb63b397eb" containerName="cleanup" Oct 30 20:04:15.417585 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.417545 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="356beeac-354b-475d-9e61-5fd58359ad64" containerName="revoke-keys" Oct 30 20:04:15.423070 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.423044 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:15.437978 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.437951 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr"] Oct 30 20:04:15.448215 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.448195 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:15.448349 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.448248 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:04:15.448424 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.448378 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:04:15.454380 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.454363 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vpw8\"" Oct 30 20:04:15.534423 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.534402 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c51798-894a-4a93-929f-5f368859b47a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:15.534528 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.534428 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c51798-894a-4a93-929f-5f368859b47a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:15.534528 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.534458 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsld\" (UniqueName: \"kubernetes.io/projected/35c51798-894a-4a93-929f-5f368859b47a-kube-api-access-wjsld\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:15.635792 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.635771 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c51798-894a-4a93-929f-5f368859b47a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:15.635901 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.635795 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c51798-894a-4a93-929f-5f368859b47a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:15.635901 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.635825 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsld\" (UniqueName: \"kubernetes.io/projected/35c51798-894a-4a93-929f-5f368859b47a-kube-api-access-wjsld\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:15.636414 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.636395 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c51798-894a-4a93-929f-5f368859b47a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:15.638281 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.638262 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c51798-894a-4a93-929f-5f368859b47a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:15.657587 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.657560 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsld\" (UniqueName: \"kubernetes.io/projected/35c51798-894a-4a93-929f-5f368859b47a-kube-api-access-wjsld\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:15.738488 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:15.738465 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:16.043929 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:16.043902 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr"] Oct 30 20:04:16.044484 ip-10-0-138-21 kubenswrapper[4904]: W1030 20:04:16.044459 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c51798_894a_4a93_929f_5f368859b47a.slice/crio-3ae61b634f3b40a573aace2d6d446101d90ce20b82b2ea04c71deedd0ae1cc98 WatchSource:0}: Error finding container 3ae61b634f3b40a573aace2d6d446101d90ce20b82b2ea04c71deedd0ae1cc98: Status 404 returned error can't find the container with id 3ae61b634f3b40a573aace2d6d446101d90ce20b82b2ea04c71deedd0ae1cc98 Oct 30 20:04:16.107604 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:16.107575 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" event={"ID":"35c51798-894a-4a93-929f-5f368859b47a","Type":"ContainerStarted","Data":"3ae61b634f3b40a573aace2d6d446101d90ce20b82b2ea04c71deedd0ae1cc98"} Oct 30 20:04:17.115884 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:17.115800 4904 generic.go:356] "Generic (PLEG): container finished" podID="35c51798-894a-4a93-929f-5f368859b47a" containerID="e5e9a270b0f2e4ff43dcab1f9a0c753994277ecd943fbe8a69a9a75622bfdefd" exitCode=0 Oct 30 20:04:17.116280 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:17.115886 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" event={"ID":"35c51798-894a-4a93-929f-5f368859b47a","Type":"ContainerDied","Data":"e5e9a270b0f2e4ff43dcab1f9a0c753994277ecd943fbe8a69a9a75622bfdefd"} Oct 30 20:04:18.263634 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:18.263608 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:18.361072 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:18.361045 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c51798-894a-4a93-929f-5f368859b47a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c51798-894a-4a93-929f-5f368859b47a-maas-api-xks-ca\") pod \"35c51798-894a-4a93-929f-5f368859b47a\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " Oct 30 20:04:18.361236 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:18.361161 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c51798-894a-4a93-929f-5f368859b47a-kube-api-access-wjsld\" (UniqueName: \"kubernetes.io/projected/35c51798-894a-4a93-929f-5f368859b47a-kube-api-access-wjsld\") pod \"35c51798-894a-4a93-929f-5f368859b47a\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " Oct 30 20:04:18.361236 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:18.361203 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c51798-894a-4a93-929f-5f368859b47a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c51798-894a-4a93-929f-5f368859b47a-maas-api-openshift-ca\") pod \"35c51798-894a-4a93-929f-5f368859b47a\" (UID: \"35c51798-894a-4a93-929f-5f368859b47a\") " Oct 30 20:04:18.361651 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:18.361586 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c51798-894a-4a93-929f-5f368859b47a-maas-api-openshift-ca" pod "35c51798-894a-4a93-929f-5f368859b47a" (UID: "35c51798-894a-4a93-929f-5f368859b47a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:18.363148 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:18.363125 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c51798-894a-4a93-929f-5f368859b47a-maas-api-xks-ca" pod "35c51798-894a-4a93-929f-5f368859b47a" (UID: "35c51798-894a-4a93-929f-5f368859b47a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:18.363784 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:18.363754 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c51798-894a-4a93-929f-5f368859b47a-kube-api-access-wjsld" pod "35c51798-894a-4a93-929f-5f368859b47a" (UID: "35c51798-894a-4a93-929f-5f368859b47a"). InnerVolumeSpecName "kube-api-access-wjsld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:18.462809 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:18.462783 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c51798-894a-4a93-929f-5f368859b47a-maas-api-xks-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:04:18.462958 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:18.462813 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjsld\" (UniqueName: \"kubernetes.io/projected/35c51798-894a-4a93-929f-5f368859b47a-kube-api-access-wjsld\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:04:18.462958 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:18.462825 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c51798-894a-4a93-929f-5f368859b47a-maas-api-openshift-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.129916 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:19.129887 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" Oct 30 20:04:19.130085 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:19.129915 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr" event={"ID":"35c51798-894a-4a93-929f-5f368859b47a","Type":"ContainerDied","Data":"3ae61b634f3b40a573aace2d6d446101d90ce20b82b2ea04c71deedd0ae1cc98"} Oct 30 20:04:19.130085 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:19.129961 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ae61b634f3b40a573aace2d6d446101d90ce20b82b2ea04c71deedd0ae1cc98" Oct 30 20:04:24.511265 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:04:24.511231 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:04:33.117385 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:33.117346 4904 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr"] Oct 30 20:04:33.130432 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:33.130396 4904 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31a039-fz4zr"] Oct 30 20:04:33.220507 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:33.220475 4904 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c51798-894a-4a93-929f-5f368859b47a" path="/var/lib/kubelet/pods/35c51798-894a-4a93-929f-5f368859b47a/volumes" Oct 30 20:04:54.508630 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:04:54.508548 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:04:57.375600 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.375567 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx"] Oct 30 20:04:57.375998 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.375947 4904 cpu_manager.go:414] "containerMap: removing container" podUID="35c51798-894a-4a93-929f-5f368859b47a" containerName="revoke-keys" Oct 30 20:04:57.375998 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.375957 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c51798-894a-4a93-929f-5f368859b47a" containerName="revoke-keys" Oct 30 20:04:57.376073 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.376040 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c51798-894a-4a93-929f-5f368859b47a" containerName="revoke-keys" Oct 30 20:04:57.379508 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.379492 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.397040 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.397018 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx"] Oct 30 20:04:57.407874 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.407849 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:04:57.407997 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.407863 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:57.407997 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.407865 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:04:57.411212 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.411197 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vpw8\"" Oct 30 20:04:57.436604 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.436578 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9v4w\" (UniqueName: \"kubernetes.io/projected/f4344ed8-f420-4f54-a804-5afa8b09d32a-kube-api-access-c9v4w\") pod \"maas-api-revoke-sub-f03629db-tk7fx\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.436723 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.436706 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tk7fx\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.436781 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.436745 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tk7fx\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.537051 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.537031 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9v4w\" (UniqueName: \"kubernetes.io/projected/f4344ed8-f420-4f54-a804-5afa8b09d32a-kube-api-access-c9v4w\") pod \"maas-api-revoke-sub-f03629db-tk7fx\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.537190 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.537096 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tk7fx\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.537190 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.537141 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tk7fx\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.537780 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.537759 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tk7fx\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.539780 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.539756 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tk7fx\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.560675 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.560646 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9v4w\" (UniqueName: \"kubernetes.io/projected/f4344ed8-f420-4f54-a804-5afa8b09d32a-kube-api-access-c9v4w\") pod \"maas-api-revoke-sub-f03629db-tk7fx\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.692422 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.692398 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:04:57.994588 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:57.994559 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx"] Oct 30 20:04:57.995624 ip-10-0-138-21 kubenswrapper[4904]: W1030 20:04:57.995599 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4344ed8_f420_4f54_a804_5afa8b09d32a.slice/crio-72c4f57f4e9db24a364e887e4c14bee9e38215f4405232d32cceae9810f53433 WatchSource:0}: Error finding container 72c4f57f4e9db24a364e887e4c14bee9e38215f4405232d32cceae9810f53433: Status 404 returned error can't find the container with id 72c4f57f4e9db24a364e887e4c14bee9e38215f4405232d32cceae9810f53433 Oct 30 20:04:58.400963 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:58.400932 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" event={"ID":"f4344ed8-f420-4f54-a804-5afa8b09d32a","Type":"ContainerStarted","Data":"5572b5f59f5245fd763dd64f3fc82b37ce574c1977f265e77e5215c3f199b7ce"} Oct 30 20:04:58.401315 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:58.400995 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" event={"ID":"f4344ed8-f420-4f54-a804-5afa8b09d32a","Type":"ContainerStarted","Data":"72c4f57f4e9db24a364e887e4c14bee9e38215f4405232d32cceae9810f53433"} Oct 30 20:04:58.444895 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:58.444851 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" podStartSLOduration=1.444839337 podStartE2EDuration="1.444839337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:58.44249066 +0000 UTC m=+2659.356095288" watchObservedRunningTime="2026-10-30 20:04:58.444839337 +0000 UTC m=+2659.358443954" Oct 30 20:04:59.409419 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:59.409390 4904 generic.go:356] "Generic (PLEG): container finished" podID="f4344ed8-f420-4f54-a804-5afa8b09d32a" containerID="5572b5f59f5245fd763dd64f3fc82b37ce574c1977f265e77e5215c3f199b7ce" exitCode=0 Oct 30 20:04:59.409788 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:04:59.409449 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" event={"ID":"f4344ed8-f420-4f54-a804-5afa8b09d32a","Type":"ContainerDied","Data":"5572b5f59f5245fd763dd64f3fc82b37ce574c1977f265e77e5215c3f199b7ce"} Oct 30 20:05:00.558943 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:00.558917 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:05:00.560968 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:00.560943 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4344ed8-f420-4f54-a804-5afa8b09d32a-kube-api-access-c9v4w\" (UniqueName: \"kubernetes.io/projected/f4344ed8-f420-4f54-a804-5afa8b09d32a-kube-api-access-c9v4w\") pod \"f4344ed8-f420-4f54-a804-5afa8b09d32a\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " Oct 30 20:05:00.561023 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:00.560992 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-xks-ca\") pod \"f4344ed8-f420-4f54-a804-5afa8b09d32a\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " Oct 30 20:05:00.561090 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:00.561070 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-openshift-ca\") pod \"f4344ed8-f420-4f54-a804-5afa8b09d32a\" (UID: \"f4344ed8-f420-4f54-a804-5afa8b09d32a\") " Oct 30 20:05:00.561368 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:00.561341 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-openshift-ca" pod "f4344ed8-f420-4f54-a804-5afa8b09d32a" (UID: "f4344ed8-f420-4f54-a804-5afa8b09d32a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:00.563087 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:00.563070 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-xks-ca" pod "f4344ed8-f420-4f54-a804-5afa8b09d32a" (UID: "f4344ed8-f420-4f54-a804-5afa8b09d32a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:00.563191 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:00.563159 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4344ed8-f420-4f54-a804-5afa8b09d32a-kube-api-access-c9v4w" pod "f4344ed8-f420-4f54-a804-5afa8b09d32a" (UID: "f4344ed8-f420-4f54-a804-5afa8b09d32a"). InnerVolumeSpecName "kube-api-access-c9v4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:00.661890 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:00.661863 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-openshift-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:05:00.661890 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:00.661884 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9v4w\" (UniqueName: \"kubernetes.io/projected/f4344ed8-f420-4f54-a804-5afa8b09d32a-kube-api-access-c9v4w\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:05:00.662037 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:00.661894 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4344ed8-f420-4f54-a804-5afa8b09d32a-maas-api-xks-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:05:01.422306 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:01.422266 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" event={"ID":"f4344ed8-f420-4f54-a804-5afa8b09d32a","Type":"ContainerDied","Data":"72c4f57f4e9db24a364e887e4c14bee9e38215f4405232d32cceae9810f53433"} Oct 30 20:05:01.422306 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:01.422287 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tk7fx" Oct 30 20:05:01.422492 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:01.422345 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72c4f57f4e9db24a364e887e4c14bee9e38215f4405232d32cceae9810f53433" Oct 30 20:05:14.755717 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.755690 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm"] Oct 30 20:05:14.756145 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.756132 4904 cpu_manager.go:414] "containerMap: removing container" podUID="f4344ed8-f420-4f54-a804-5afa8b09d32a" containerName="revoke-keys" Oct 30 20:05:14.756200 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.756146 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4344ed8-f420-4f54-a804-5afa8b09d32a" containerName="revoke-keys" Oct 30 20:05:14.756247 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.756223 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4344ed8-f420-4f54-a804-5afa8b09d32a" containerName="revoke-keys" Oct 30 20:05:14.759483 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.759467 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:14.777686 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.777664 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm"] Oct 30 20:05:14.787476 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.787453 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:05:14.787698 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.787674 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:05:14.787864 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.787732 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:05:14.789424 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.789407 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vpw8\"" Oct 30 20:05:14.808016 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.807995 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zzszm\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:14.808141 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.808059 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llz42\" (UniqueName: \"kubernetes.io/projected/5eafb185-c039-4f6d-8beb-7f7e0346b25b-kube-api-access-llz42\") pod \"maas-api-revoke-sub-0165d7d2-zzszm\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:14.808192 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.808139 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zzszm\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:14.909497 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.909470 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zzszm\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:14.909599 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.909503 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llz42\" (UniqueName: \"kubernetes.io/projected/5eafb185-c039-4f6d-8beb-7f7e0346b25b-kube-api-access-llz42\") pod \"maas-api-revoke-sub-0165d7d2-zzszm\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:14.909649 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.909630 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zzszm\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:14.910181 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.910154 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zzszm\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:14.912059 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.912039 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zzszm\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:14.930091 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:14.930069 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llz42\" (UniqueName: \"kubernetes.io/projected/5eafb185-c039-4f6d-8beb-7f7e0346b25b-kube-api-access-llz42\") pod \"maas-api-revoke-sub-0165d7d2-zzszm\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:15.072831 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:15.072780 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:15.374736 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:15.374709 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm"] Oct 30 20:05:15.376030 ip-10-0-138-21 kubenswrapper[4904]: W1030 20:05:15.376007 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eafb185_c039_4f6d_8beb_7f7e0346b25b.slice/crio-f5798c4ff1695c8857c7f73760a6a25618d197d4875626bfe2eaa2d12ae71be4 WatchSource:0}: Error finding container f5798c4ff1695c8857c7f73760a6a25618d197d4875626bfe2eaa2d12ae71be4: Status 404 returned error can't find the container with id f5798c4ff1695c8857c7f73760a6a25618d197d4875626bfe2eaa2d12ae71be4 Oct 30 20:05:15.517550 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:15.517521 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" event={"ID":"5eafb185-c039-4f6d-8beb-7f7e0346b25b","Type":"ContainerStarted","Data":"f5798c4ff1695c8857c7f73760a6a25618d197d4875626bfe2eaa2d12ae71be4"} Oct 30 20:05:16.525959 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:16.525925 4904 generic.go:356] "Generic (PLEG): container finished" podID="5eafb185-c039-4f6d-8beb-7f7e0346b25b" containerID="26e75624c8d9679252e1621da937cbfa6a71af6c85a197a9968ef5f5eaeacc0f" exitCode=0 Oct 30 20:05:16.526334 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:16.526012 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" event={"ID":"5eafb185-c039-4f6d-8beb-7f7e0346b25b","Type":"ContainerDied","Data":"26e75624c8d9679252e1621da937cbfa6a71af6c85a197a9968ef5f5eaeacc0f"} Oct 30 20:05:17.680143 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:17.680120 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:17.734796 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:17.734774 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-openshift-ca\") pod \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " Oct 30 20:05:17.734920 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:17.734874 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-xks-ca\") pod \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " Oct 30 20:05:17.734981 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:17.734942 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eafb185-c039-4f6d-8beb-7f7e0346b25b-kube-api-access-llz42\" (UniqueName: \"kubernetes.io/projected/5eafb185-c039-4f6d-8beb-7f7e0346b25b-kube-api-access-llz42\") pod \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\" (UID: \"5eafb185-c039-4f6d-8beb-7f7e0346b25b\") " Oct 30 20:05:17.735208 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:17.735180 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-openshift-ca" pod "5eafb185-c039-4f6d-8beb-7f7e0346b25b" (UID: "5eafb185-c039-4f6d-8beb-7f7e0346b25b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:17.737128 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:17.737077 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-xks-ca" pod "5eafb185-c039-4f6d-8beb-7f7e0346b25b" (UID: "5eafb185-c039-4f6d-8beb-7f7e0346b25b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:17.737208 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:17.737131 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eafb185-c039-4f6d-8beb-7f7e0346b25b-kube-api-access-llz42" pod "5eafb185-c039-4f6d-8beb-7f7e0346b25b" (UID: "5eafb185-c039-4f6d-8beb-7f7e0346b25b"). InnerVolumeSpecName "kube-api-access-llz42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:17.835925 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:17.835873 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-openshift-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:05:17.835925 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:17.835898 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eafb185-c039-4f6d-8beb-7f7e0346b25b-maas-api-xks-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:05:17.835925 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:17.835908 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llz42\" (UniqueName: \"kubernetes.io/projected/5eafb185-c039-4f6d-8beb-7f7e0346b25b-kube-api-access-llz42\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:05:18.540049 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:18.540021 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" Oct 30 20:05:18.540262 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:18.540020 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zzszm" event={"ID":"5eafb185-c039-4f6d-8beb-7f7e0346b25b","Type":"ContainerDied","Data":"f5798c4ff1695c8857c7f73760a6a25618d197d4875626bfe2eaa2d12ae71be4"} Oct 30 20:05:18.540262 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:18.540151 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5798c4ff1695c8857c7f73760a6a25618d197d4875626bfe2eaa2d12ae71be4" Oct 30 20:05:24.509326 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:05:24.509289 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:05:25.697726 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.697696 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw"] Oct 30 20:05:25.698227 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.698127 4904 cpu_manager.go:414] "containerMap: removing container" podUID="5eafb185-c039-4f6d-8beb-7f7e0346b25b" containerName="revoke-keys" Oct 30 20:05:25.698227 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.698140 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eafb185-c039-4f6d-8beb-7f7e0346b25b" containerName="revoke-keys" Oct 30 20:05:25.698227 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.698208 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eafb185-c039-4f6d-8beb-7f7e0346b25b" containerName="revoke-keys" Oct 30 20:05:25.701209 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.701185 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:25.719194 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.719170 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw"] Oct 30 20:05:25.729667 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.729645 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:05:25.729782 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.729677 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:05:25.729782 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.729679 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:05:25.731737 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.731714 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vpw8\"" Oct 30 20:05:25.800573 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.800540 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-l84mw\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:25.800743 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.800700 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-l84mw\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:25.800786 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.800755 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sh75\" (UniqueName: \"kubernetes.io/projected/f74fce85-76d1-4f55-b548-5c8091371685-kube-api-access-9sh75\") pod \"maas-api-revoke-sub-6b19338f-l84mw\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:25.902154 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.902095 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-l84mw\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:25.902315 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.902218 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-l84mw\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:25.902315 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.902245 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sh75\" (UniqueName: \"kubernetes.io/projected/f74fce85-76d1-4f55-b548-5c8091371685-kube-api-access-9sh75\") pod \"maas-api-revoke-sub-6b19338f-l84mw\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:25.903001 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.902976 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-l84mw\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:25.904533 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.904512 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-l84mw\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:25.924114 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:25.924072 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sh75\" (UniqueName: \"kubernetes.io/projected/f74fce85-76d1-4f55-b548-5c8091371685-kube-api-access-9sh75\") pod \"maas-api-revoke-sub-6b19338f-l84mw\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:26.015278 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:26.015213 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:26.344524 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:26.344484 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw"] Oct 30 20:05:26.345685 ip-10-0-138-21 kubenswrapper[4904]: W1030 20:05:26.345655 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74fce85_76d1_4f55_b548_5c8091371685.slice/crio-395e1fcbddeb243b3112e0a537f31e225c873aaea8f3521838a65ffecd102a0d WatchSource:0}: Error finding container 395e1fcbddeb243b3112e0a537f31e225c873aaea8f3521838a65ffecd102a0d: Status 404 returned error can't find the container with id 395e1fcbddeb243b3112e0a537f31e225c873aaea8f3521838a65ffecd102a0d Oct 30 20:05:26.591507 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:26.591466 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" event={"ID":"f74fce85-76d1-4f55-b548-5c8091371685","Type":"ContainerStarted","Data":"395e1fcbddeb243b3112e0a537f31e225c873aaea8f3521838a65ffecd102a0d"} Oct 30 20:05:27.598789 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:27.598704 4904 generic.go:356] "Generic (PLEG): container finished" podID="f74fce85-76d1-4f55-b548-5c8091371685" containerID="eb81bc009e451edb46ec0e3972478f00bfde1e96613f2bd7bb09c57974626a5f" exitCode=0 Oct 30 20:05:27.598789 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:27.598753 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" event={"ID":"f74fce85-76d1-4f55-b548-5c8091371685","Type":"ContainerDied","Data":"eb81bc009e451edb46ec0e3972478f00bfde1e96613f2bd7bb09c57974626a5f"} Oct 30 20:05:28.744574 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:28.744548 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:28.828247 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:28.828223 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-xks-ca\") pod \"f74fce85-76d1-4f55-b548-5c8091371685\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " Oct 30 20:05:28.828388 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:28.828292 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74fce85-76d1-4f55-b548-5c8091371685-kube-api-access-9sh75\" (UniqueName: \"kubernetes.io/projected/f74fce85-76d1-4f55-b548-5c8091371685-kube-api-access-9sh75\") pod \"f74fce85-76d1-4f55-b548-5c8091371685\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " Oct 30 20:05:28.828388 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:28.828375 4904 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-openshift-ca\") pod \"f74fce85-76d1-4f55-b548-5c8091371685\" (UID: \"f74fce85-76d1-4f55-b548-5c8091371685\") " Oct 30 20:05:28.828736 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:28.828707 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-openshift-ca" pod "f74fce85-76d1-4f55-b548-5c8091371685" (UID: "f74fce85-76d1-4f55-b548-5c8091371685"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:28.830500 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:28.830474 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74fce85-76d1-4f55-b548-5c8091371685-kube-api-access-9sh75" pod "f74fce85-76d1-4f55-b548-5c8091371685" (UID: "f74fce85-76d1-4f55-b548-5c8091371685"). InnerVolumeSpecName "kube-api-access-9sh75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:28.830660 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:28.830640 4904 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-xks-ca" pod "f74fce85-76d1-4f55-b548-5c8091371685" (UID: "f74fce85-76d1-4f55-b548-5c8091371685"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:28.929921 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:28.929892 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-openshift-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:05:28.929921 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:28.929917 4904 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74fce85-76d1-4f55-b548-5c8091371685-maas-api-xks-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:05:28.930142 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:28.929926 4904 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sh75\" (UniqueName: \"kubernetes.io/projected/f74fce85-76d1-4f55-b548-5c8091371685-kube-api-access-9sh75\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\"" Oct 30 20:05:29.612554 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:29.612475 4904 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" Oct 30 20:05:29.612554 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:29.612517 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-l84mw" event={"ID":"f74fce85-76d1-4f55-b548-5c8091371685","Type":"ContainerDied","Data":"395e1fcbddeb243b3112e0a537f31e225c873aaea8f3521838a65ffecd102a0d"} Oct 30 20:05:29.612795 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:29.612563 4904 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="395e1fcbddeb243b3112e0a537f31e225c873aaea8f3521838a65ffecd102a0d" Oct 30 20:05:37.083186 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:37.083153 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54554668db-6fwsb_9ead0538-3773-4680-b10f-038113f65fcf/authorino/0.log" Oct 30 20:05:39.895923 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:39.895895 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 20:05:39.896419 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:39.896307 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5ckd_550c652c-4e47-487a-ae4e-1324b8d44cf4/extract/0.log" Oct 30 20:05:39.912795 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:39.912766 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 20:05:39.912919 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:39.912893 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 20:05:47.760629 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:47.760594 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9kdnq_b0dc02fe-b7e5-419e-80af-d0c9df5b6742/manager/0.log" Oct 30 20:05:47.962658 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:47.962623 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fl9ss_355589dc-40b8-4ad2-8566-9f7eb2ce069f/manager/0.log" Oct 30 20:05:48.171548 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:48.171519 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-4lh4b_c9e60db3-d94e-40a8-9ad0-8abe2148fe84/manager/0.log" Oct 30 20:05:48.716407 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:48.716384 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-prj9w_c5898fda-c3ce-4907-a685-058d32307825/server/0.log" Oct 30 20:05:48.905246 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:48.905217 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-795kr_1ad4552e-0da1-49db-80b8-fb4573db1d60/manager/0.log" Oct 30 20:05:50.284172 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:50.284134 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zzszm_5eafb185-c039-4f6d-8beb-7f7e0346b25b/revoke-keys/0.log" Oct 30 20:05:50.875258 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:50.875229 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-l84mw_f74fce85-76d1-4f55-b548-5c8091371685/revoke-keys/0.log" Oct 30 20:05:51.274540 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:51.274514 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-tk7fx_f4344ed8-f420-4f54-a804-5afa8b09d32a/revoke-keys/0.log" Oct 30 20:05:53.007203 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:53.007171 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54554668db-6fwsb_9ead0538-3773-4680-b10f-038113f65fcf/authorino/0.log" Oct 30 20:05:53.265164 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:53.265083 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-whljk_84ea9711-daf8-4c43-81c7-8a31cc88d45f/manager/0.log" Oct 30 20:05:53.459032 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:53.459001 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-t8d5f_bcbc334e-5d81-4772-abc0-4aa196f064b6/manager/0.log" Oct 30 20:05:53.919956 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:53.919926 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hwhj6_336f7de4-e1c6-4e67-8252-9f5299137355/manager/0.log" Oct 30 20:05:54.387580 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:54.387545 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-lln9b_b85b9f89-71c2-4664-aaad-29a15ef6f6b2/manager/0.log" Oct 30 20:05:54.512976 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:05:54.512942 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:05:55.164895 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:55.164869 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fqv2lt_e980f2ff-8bfb-4fbf-9bcc-d72274125bc6/istio-proxy/0.log" Oct 30 20:05:56.167656 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:56.167619 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8k4b6_8da395a9-9be0-46ba-b397-ee54354bf861/istio-proxy/0.log" Oct 30 20:05:56.964992 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:56.964956 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ggdsj_0c8a2a4b-6240-40fa-a873-d2300e317a2b/istio-proxy/1.log" Oct 30 20:05:57.734095 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:57.734066 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x4dsn_2c012900-326d-42d5-8076-3bbbdbb1e852/istio-proxy/0.log" Oct 30 20:05:57.922003 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:57.921973 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk_c06db63b-4ed7-499f-92e8-9f8ebd75598b/istio-proxy/0.log" Oct 30 20:05:58.491915 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:58.491887 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56bc986487-ljfn2_605b308f-b6a9-427f-a365-c0f8b110da64/router/0.log" Oct 30 20:05:58.947827 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:05:58.947797 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ggdsj_0c8a2a4b-6240-40fa-a873-d2300e317a2b/istio-proxy/0.log" Oct 30 20:06:01.891419 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:01.891391 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54554668db-6fwsb_9ead0538-3773-4680-b10f-038113f65fcf/authorino/0.log" Oct 30 20:06:11.431981 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:11.431948 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9kdnq_b0dc02fe-b7e5-419e-80af-d0c9df5b6742/manager/0.log" Oct 30 20:06:11.649618 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:11.649590 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fl9ss_355589dc-40b8-4ad2-8566-9f7eb2ce069f/manager/0.log" Oct 30 20:06:11.861453 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:11.861374 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-4lh4b_c9e60db3-d94e-40a8-9ad0-8abe2148fe84/manager/0.log" Oct 30 20:06:12.311502 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:12.311474 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-prj9w_c5898fda-c3ce-4907-a685-058d32307825/server/0.log" Oct 30 20:06:12.517027 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:12.516997 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-795kr_1ad4552e-0da1-49db-80b8-fb4573db1d60/manager/0.log" Oct 30 20:06:13.695893 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:13.695865 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zzszm_5eafb185-c039-4f6d-8beb-7f7e0346b25b/revoke-keys/0.log" Oct 30 20:06:14.083930 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:14.083853 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-l84mw_f74fce85-76d1-4f55-b548-5c8091371685/revoke-keys/0.log" Oct 30 20:06:14.466342 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:14.466315 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-tk7fx_f4344ed8-f420-4f54-a804-5afa8b09d32a/revoke-keys/0.log" Oct 30 20:06:15.778003 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:15.777972 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54554668db-6fwsb_9ead0538-3773-4680-b10f-038113f65fcf/authorino/0.log" Oct 30 20:06:16.007887 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:16.007855 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-whljk_84ea9711-daf8-4c43-81c7-8a31cc88d45f/manager/0.log" Oct 30 20:06:16.204779 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:16.204750 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-t8d5f_bcbc334e-5d81-4772-abc0-4aa196f064b6/manager/0.log" Oct 30 20:06:16.596525 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:16.596455 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hwhj6_336f7de4-e1c6-4e67-8252-9f5299137355/manager/0.log" Oct 30 20:06:16.981580 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:16.981549 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-lln9b_b85b9f89-71c2-4664-aaad-29a15ef6f6b2/manager/0.log" Oct 30 20:06:17.573277 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:17.573246 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fqv2lt_e980f2ff-8bfb-4fbf-9bcc-d72274125bc6/istio-proxy/0.log" Oct 30 20:06:18.600226 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:18.600196 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8k4b6_8da395a9-9be0-46ba-b397-ee54354bf861/istio-proxy/0.log" Oct 30 20:06:19.405485 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:19.405396 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ggdsj_0c8a2a4b-6240-40fa-a873-d2300e317a2b/istio-proxy/1.log" Oct 30 20:06:20.190872 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:20.190846 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x4dsn_2c012900-326d-42d5-8076-3bbbdbb1e852/istio-proxy/0.log" Oct 30 20:06:20.386665 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:20.386633 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk_c06db63b-4ed7-499f-92e8-9f8ebd75598b/istio-proxy/0.log" Oct 30 20:06:20.952697 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:06:20.952668 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56bc986487-ljfn2_605b308f-b6a9-427f-a365-c0f8b110da64/router/0.log" Oct 30 20:06:24.508315 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:06:24.508281 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:06:54.510250 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:06:54.510215 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:07:24.509973 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:07:24.509932 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:07:47.704694 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:47.704663 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vfmk2_6829a2b0-e64b-4b3c-9007-966545b88c8d/global-pull-secret-syncer/0.log" Oct 30 20:07:48.027305 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:48.027232 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-c6bdc_b07766d9-9884-4f58-9c94-e59540953a06/connection-checker/0.log" Oct 30 20:07:48.133734 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:48.133703 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gt86s_e6811b53-7458-4b8c-b4f9-9f46fb029eea/connection-checker/0.log" Oct 30 20:07:48.245485 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:48.245458 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qgdzh_9f1f8b8c-7daa-4eb9-8e5e-4c73b44359b1/connection-checker/0.log" Oct 30 20:07:48.663589 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:48.663562 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ttpmm_1d809989-435e-4d38-807e-4bdbe414cf32/konnectivity-agent/0.log" Oct 30 20:07:48.879572 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:48.879545 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-21.ec2.internal_62fcc3a86a753eb9d656b3c26d634e28/haproxy/0.log" Oct 30 20:07:54.517533 ip-10-0-138-21 kubenswrapper[4904]: E1030 20:07:54.517450 4904 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l55ql" containerName="extractor" Oct 30 20:07:54.956115 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:54.956063 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54554668db-6fwsb_9ead0538-3773-4680-b10f-038113f65fcf/authorino/0.log" Oct 30 20:07:55.042706 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:55.042676 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-whljk_84ea9711-daf8-4c43-81c7-8a31cc88d45f/manager/0.log" Oct 30 20:07:55.149997 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:55.149972 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-t8d5f_bcbc334e-5d81-4772-abc0-4aa196f064b6/manager/0.log" Oct 30 20:07:55.425076 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:55.425046 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hwhj6_336f7de4-e1c6-4e67-8252-9f5299137355/manager/0.log" Oct 30 20:07:55.697439 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:55.697361 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-lln9b_b85b9f89-71c2-4664-aaad-29a15ef6f6b2/manager/0.log" Oct 30 20:07:57.302611 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.302582 4904 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw"] Oct 30 20:07:57.303188 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.303027 4904 cpu_manager.go:414] "containerMap: removing container" podUID="f74fce85-76d1-4f55-b548-5c8091371685" containerName="revoke-keys" Oct 30 20:07:57.303188 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.303042 4904 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74fce85-76d1-4f55-b548-5c8091371685" containerName="revoke-keys" Oct 30 20:07:57.303188 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.303151 4904 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74fce85-76d1-4f55-b548-5c8091371685" containerName="revoke-keys" Oct 30 20:07:57.306396 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.306376 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.328369 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.328347 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw"] Oct 30 20:07:57.424290 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.424266 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zq2q\"/\"openshift-service-ca.crt\"" Oct 30 20:07:57.424422 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.424407 4904 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9zq2q\"/\"default-dockercfg-rgcbq\"" Oct 30 20:07:57.424422 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.424415 4904 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zq2q\"/\"kube-root-ca.crt\"" Oct 30 20:07:57.462756 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.462724 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj5ml\" (UniqueName: \"kubernetes.io/projected/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-kube-api-access-dj5ml\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.462756 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.462756 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-podres\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.462902 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.462802 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-sys\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.462939 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.462898 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-lib-modules\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.462939 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.462922 4904 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-proc\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.563773 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.563718 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj5ml\" (UniqueName: \"kubernetes.io/projected/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-kube-api-access-dj5ml\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.563773 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.563748 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-podres\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.563940 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.563782 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-sys\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.563940 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.563802 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-lib-modules\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.563940 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.563817 4904 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-proc\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.563940 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.563866 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-sys\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.563940 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.563880 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-lib-modules\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.563940 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.563866 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-podres\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.563940 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.563894 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-proc\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.585227 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.585201 4904 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj5ml\" (UniqueName: \"kubernetes.io/projected/68c7dd9e-c9d8-4986-8e51-ae3048cd62b0-kube-api-access-dj5ml\") pod \"perf-node-gather-daemonset-6pdpw\" (UID: \"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.618843 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.618825 4904 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:57.924206 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.924174 4904 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw"] Oct 30 20:07:57.925281 ip-10-0-138-21 kubenswrapper[4904]: W1030 20:07:57.925246 4904 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod68c7dd9e_c9d8_4986_8e51_ae3048cd62b0.slice/crio-d39e45373540abed059de5fd4c1b5d90400e4eed7703bccedf4a24aaf28ef106 WatchSource:0}: Error finding container d39e45373540abed059de5fd4c1b5d90400e4eed7703bccedf4a24aaf28ef106: Status 404 returned error can't find the container with id d39e45373540abed059de5fd4c1b5d90400e4eed7703bccedf4a24aaf28ef106 Oct 30 20:07:57.995126 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:57.995097 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-fgx9q_d7dfc7a1-6b86-471f-b85b-d5ea756db9cb/cluster-monitoring-operator/0.log" Oct 30 20:07:58.507991 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:58.507959 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" event={"ID":"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0","Type":"ContainerStarted","Data":"1761296e1a817965829264f755ce7b4e3882167dc3f156a2425eb0e07d04ecd0"} Oct 30 20:07:58.508411 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:58.508037 4904 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" event={"ID":"68c7dd9e-c9d8-4986-8e51-ae3048cd62b0","Type":"ContainerStarted","Data":"d39e45373540abed059de5fd4c1b5d90400e4eed7703bccedf4a24aaf28ef106"} Oct 30 20:07:58.508411 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:58.508130 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:07:58.552594 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:58.552553 4904 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" podStartSLOduration=1.5525409639999999 podStartE2EDuration="1.552540964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:58.55033705 +0000 UTC m=+2839.463941667" watchObservedRunningTime="2026-10-30 20:07:58.552540964 +0000 UTC m=+2839.466145604" Oct 30 20:07:58.970418 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:58.970388 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcwl9_029546f8-af87-4430-9de1-725b5e3389cf/node-exporter/0.log" Oct 30 20:07:59.085481 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:59.085453 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcwl9_029546f8-af87-4430-9de1-725b5e3389cf/kube-rbac-proxy/0.log" Oct 30 20:07:59.190516 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:07:59.190492 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcwl9_029546f8-af87-4430-9de1-725b5e3389cf/init-textfile/0.log" Oct 30 20:08:03.245788 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:03.245754 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-8ns8s_70b0df95-c5a5-48b5-b670-8382137780ea/networking-console-plugin/0.log" Oct 30 20:08:04.180426 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:04.180393 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/2.log" Oct 30 20:08:04.187991 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:04.187963 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9g8dh_1688cfe7-e0c1-4e07-ae2f-cc42f9805ed0/console-operator/3.log" Oct 30 20:08:04.525623 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:04.525563 4904 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-6pdpw" Oct 30 20:08:06.271623 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:06.271599 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fj5bt_c2f5e9c8-28aa-4956-a3ee-811d5577ec5d/volume-data-source-validator/0.log" Oct 30 20:08:08.499458 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:08.499425 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-77k4g_044847a9-7c73-421b-98ae-50bbb505b30a/dns/0.log" Oct 30 20:08:08.597234 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:08.597208 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-77k4g_044847a9-7c73-421b-98ae-50bbb505b30a/kube-rbac-proxy/0.log" Oct 30 20:08:09.135257 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:09.135229 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5lmmj_5d09042e-9b40-41d8-803d-8d4a0bbf0fd1/dns-node-resolver/0.log" Oct 30 20:08:09.656365 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:09.656331 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fqv2lt_e980f2ff-8bfb-4fbf-9bcc-d72274125bc6/istio-proxy/0.log" Oct 30 20:08:10.202743 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:10.202711 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8k4b6_8da395a9-9be0-46ba-b397-ee54354bf861/istio-proxy/0.log" Oct 30 20:08:10.447788 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:10.447757 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ggdsj_0c8a2a4b-6240-40fa-a873-d2300e317a2b/istio-proxy/0.log" Oct 30 20:08:10.450987 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:10.450962 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ggdsj_0c8a2a4b-6240-40fa-a873-d2300e317a2b/istio-proxy/1.log" Oct 30 20:08:10.501882 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:10.501821 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jsp6k_a94f5874-53f0-4be0-bb77-ae6d535356ef/node-ca/0.log" Oct 30 20:08:10.668195 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:10.668169 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x4dsn_2c012900-326d-42d5-8076-3bbbdbb1e852/istio-proxy/0.log" Oct 30 20:08:10.723553 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:10.723530 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-trgsk_c06db63b-4ed7-499f-92e8-9f8ebd75598b/istio-proxy/0.log" Oct 30 20:08:10.948693 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:10.948674 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56bc986487-ljfn2_605b308f-b6a9-427f-a365-c0f8b110da64/router/0.log" Oct 30 20:08:12.412868 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:12.412842 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fqv2lt_e980f2ff-8bfb-4fbf-9bcc-d72274125bc6/istio-proxy/0.log" Oct 30 20:08:13.738364 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:13.738332 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fqv2lt_e980f2ff-8bfb-4fbf-9bcc-d72274125bc6/istio-proxy/0.log" Oct 30 20:08:13.812610 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:13.812573 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8k4b6_8da395a9-9be0-46ba-b397-ee54354bf861/istio-proxy/0.log" Oct 30 20:08:14.344925 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:14.344888 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8k4b6_8da395a9-9be0-46ba-b397-ee54354bf861/istio-proxy/0.log" Oct 30 20:08:14.345949 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:14.345921 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ggdsj_0c8a2a4b-6240-40fa-a873-d2300e317a2b/istio-proxy/0.log" Oct 30 20:08:14.349226 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:14.349205 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ggdsj_0c8a2a4b-6240-40fa-a873-d2300e317a2b/istio-proxy/1.log" Oct 30 20:08:14.597215 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:14.597152 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ggdsj_0c8a2a4b-6240-40fa-a873-d2300e317a2b/istio-proxy/0.log" Oct 30 20:08:14.600083 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:14.600057 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ggdsj_0c8a2a4b-6240-40fa-a873-d2300e317a2b/istio-proxy/1.log" Oct 30 20:08:14.762580 ip-10-0-138-21 kubenswrapper[4904]: I1030 20:08:14.762544 4904 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x4dsn_2c012900-326d-42d5-8076-3bbbdbb1e852/istio-proxy/0.log"